Learned Autoscaling for Cloud Microservices with Multi-Armed Bandits

by   Vighnesh Sachidananda, et al.

As cloud applications shift from monolithic architectures to loosely coupled microservices, several challenges in resource management arise. Application developers are tasked with determining compute capacity needed for each microservice in an application. This allocation dictates both the cost and performance of the application and typically relies on using either machine utilization (e.g. CPU, RAM) metrics. Our approach, COLA, relies on training a contextual multi armed bandit on representative workloads for an application and uses techniques to generalize performance to unseen workloads. We evaluate workloads of varying complexity including those with a fixed rate, diurnal pattern and dynamic request distribution. Across a set of five open-source microservice applications, we compare COLA against a variety of utilization and machine learning baselines. We find COLA provides the most cost effective autoscaling solution for a desired median or tail latency target on 13 of 18 workloads. On average, clusters managed by COLA cost 25.1% fewer dollars than the next closest alternative that meets a specified target latency. We discuss several optimizations, inspired by systems and machine learning literature, we make during training to efficiently explore the space of possible microservice configurations. These optimizations enable us to train our models over the course of a few hours. The cost savings from managing a cluster with COLA result in the system paying for its own training cost within a few days.



There are no comments yet.



Using Subjective Logic to Estimate Uncertainty in Multi-Armed Bandit Problems

The multi-armed bandit problem is a classical decision-making problem wh...

Practical Calculation of Gittins Indices for Multi-armed Bandits

Gittins indices provide an optimal solution to the classical multi-armed...

No DBA? No regret! Multi-armed bandits for index tuning of analytical and HTAP workloads with provable guarantees

Automating physical database design has remained a long-term interest in...

Micky: A Cheaper Alternative for Selecting Cloud Instances

Most cloud computing optimizers explore and improve one workload at a ti...

Bao: Learning to Steer Query Optimizers

Query optimization remains one of the most challenging problems in data ...

Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

Training data for machine translation (MT) is often sourced from a multi...

INFaaS: Managed & Model-less Inference Serving

The number of applications relying on inference from machine learning mo...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.