Learned Autoscaling for Cloud Microservices with Multi-Armed Bandits
share
As cloud applications shift from monolithic architectures to loosely coupled microservices, several challenges in resource management arise. Application developers are tasked with determining compute capacity needed for each microservice in an application. This allocation dictates both the cost and performance of the application and typically relies on using either machine utilization (e.g. CPU, RAM) metrics. Our approach, COLA, relies on training a contextual multi armed bandit on representative workloads for an application and uses techniques to generalize performance to unseen workloads. We evaluate workloads of varying complexity including those with a fixed rate, diurnal pattern and dynamic request distribution. Across a set of five open-source microservice applications, we compare COLA against a variety of utilization and machine learning baselines. We find COLA provides the most cost effective autoscaling solution for a desired median or tail latency target on 13 of 18 workloads. On average, clusters managed by COLA cost 25.1% fewer dollars than the next closest alternative that meets a specified target latency. We discuss several optimizations, inspired by systems and machine learning literature, we make during training to efficiently explore the space of possible microservice configurations. These optimizations enable us to train our models over the course of a few hours. The cost savings from managing a cluster with COLA result in the system paying for its own training cost within a few days.
READ FULL TEXT