Predict-and-Critic: Accelerated End-to-End Predictive Control for Cloud Computing through Reinforcement Learning

by   Kaustubh Sridhar, et al.

Cloud computing holds the promise of reduced costs through economies of scale. To realize this promise, cloud computing vendors typically solve sequential resource allocation problems, where customer workloads are packed on shared hardware. Virtual machines (VM) form the foundation of modern cloud computing as they help logically abstract user compute from shared physical infrastructure. Traditionally, VM packing problems are solved by predicting demand, followed by a Model Predictive Control (MPC) optimization over a future horizon. We introduce an approximate formulation of an industrial VM packing problem as an MILP with soft-constraints parameterized by the predictions. Recently, predict-and-optimize (PnO) was proposed for end-to-end training of prediction models by back-propagating the cost of decisions through the optimization problem. But, PnO is unable to scale to the large prediction horizons prevalent in cloud computing. To tackle this issue, we propose the Predict-and-Critic (PnC) framework that outperforms PnO with just a two-step horizon by leveraging reinforcement learning. PnC jointly trains a prediction model and a terminal Q function that approximates cost-to-go over a long horizon, by back-propagating the cost of decisions through the optimization problem and from the future. The terminal Q function allows us to solve a much smaller two-step horizon optimization problem than the multi-step horizon necessary in PnO. We evaluate PnO and the PnC framework on two datasets, three workloads, and with disturbances not modeled in the optimization problem. We find that PnC significantly improves decision quality over PnO, even when the optimization problem is not a perfect representation of reality. We also find that hardening the soft constraints of the MILP and back-propagating through the constraints improves decision quality for both PnO and PnC.


page 1

page 2

page 3

page 4


Reinforcement Learning of the Prediction Horizon in Model Predictive Control

Model predictive control (MPC) is a powerful trajectory optimization con...

Online Resource Procurement and Allocation in a Hybrid Edge-Cloud Computing System

By acquiring cloud-like capacities at the edge of a network, edge comput...

Robust Learning-based Predictive Control for Constrained Nonlinear Systems

The integration of machine learning methods and Model Predictive Control...

Stochastic Qubit Resource Allocation for Quantum Cloud Computing

Quantum cloud computing is a promising paradigm for efficiently provisio...

CILP: Co-simulation based Imitation Learner for Dynamic Resource Provisioning in Cloud Computing Environments

Intelligent Virtual Machine (VM) provisioning is central to cost and res...

A Surrogate Objective Framework for Prediction+Optimization with Soft Constraints

Prediction+optimization is a common real-world paradigm where we have to...

Defending against Co-residence Attack in Energy-Efficient Cloud: An Optimization based Real-time Secure VM Allocation Strategy

Resource sharing among users serves as the foundation of cloud computing...

Please sign up or login with your details

Forgot password? Click here to reset