DeepAI AI Chat
Log In Sign Up

Predict-and-Critic: Accelerated End-to-End Predictive Control for Cloud Computing through Reinforcement Learning

12/02/2022
by   Kaustubh Sridhar, et al.
Amazon
University of Pennsylvania
0

Cloud computing holds the promise of reduced costs through economies of scale. To realize this promise, cloud computing vendors typically solve sequential resource allocation problems, where customer workloads are packed on shared hardware. Virtual machines (VM) form the foundation of modern cloud computing as they help logically abstract user compute from shared physical infrastructure. Traditionally, VM packing problems are solved by predicting demand, followed by a Model Predictive Control (MPC) optimization over a future horizon. We introduce an approximate formulation of an industrial VM packing problem as an MILP with soft-constraints parameterized by the predictions. Recently, predict-and-optimize (PnO) was proposed for end-to-end training of prediction models by back-propagating the cost of decisions through the optimization problem. But, PnO is unable to scale to the large prediction horizons prevalent in cloud computing. To tackle this issue, we propose the Predict-and-Critic (PnC) framework that outperforms PnO with just a two-step horizon by leveraging reinforcement learning. PnC jointly trains a prediction model and a terminal Q function that approximates cost-to-go over a long horizon, by back-propagating the cost of decisions through the optimization problem and from the future. The terminal Q function allows us to solve a much smaller two-step horizon optimization problem than the multi-step horizon necessary in PnO. We evaluate PnO and the PnC framework on two datasets, three workloads, and with disturbances not modeled in the optimization problem. We find that PnC significantly improves decision quality over PnO, even when the optimization problem is not a perfect representation of reality. We also find that hardening the soft constraints of the MILP and back-propagating through the constraints improves decision quality for both PnO and PnC.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/22/2021

Reinforcement Learning of the Prediction Horizon in Model Predictive Control

Model predictive control (MPC) is a powerful trajectory optimization con...
01/24/2020

Online Resource Procurement and Allocation in a Hybrid Edge-Cloud Computing System

By acquiring cloud-like capacities at the edge of a network, edge comput...
11/22/2019

Robust Learning-based Predictive Control for Constrained Nonlinear Systems

The integration of machine learning methods and Model Predictive Control...
10/22/2022

Stochastic Qubit Resource Allocation for Quantum Cloud Computing

Quantum cloud computing is a promising paradigm for efficiently provisio...
02/11/2023

CILP: Co-simulation based Imitation Learner for Dynamic Resource Provisioning in Cloud Computing Environments

Intelligent Virtual Machine (VM) provisioning is central to cost and res...
01/29/2021

Can Machine Learning Help in Solving Cargo Capacity Management Booking Control Problems?

Revenue management is important for carriers (e.g., airlines and railroa...
11/22/2021

A Surrogate Objective Framework for Prediction+Optimization with Soft Constraints

Prediction+optimization is a common real-world paradigm where we have to...