Stability-Constrained Markov Decision Processes Using MPC

02/02/2021

∙

In this paper, we consider solving discounted Markov Decision Processes (MDPs) under the constraint that the resulting policy is stabilizing. In practice MDPs are solved based on some form of policy approximation. We will leverage recent results proposing to use Model Predictive Control (MPC) as a structured policy in the context of Reinforcement Learning to make it possible to introduce stability requirements directly inside the MPC-based policy. This will restrict the solution of the MDP to stabilizing policies by construction. The stability theory for MPC is most mature for the undiscounted MPC case. Hence, we will first show in this paper that stable discounted MDPs can be reformulated as undiscounted ones. This observation will entail that the MPC-based policy with stability requirements will produce the optimal policy for the discounted MDP if it is stable, and the best stabilizing policy otherwise.

READ FULL TEXT

Stability-Constrained Markov Decision Processes Using MPC

Metrics for Markov Decision Processes with Infinite State Spaces

Self-Triggered Markov Decision Processes

Bridging the gap between QP-based and MPC-based RL

Safe Reinforcement Learning with Stability Safety Guarantees Using Robust MPC

Bayesian regularization of empirical MDPs

Similarity metrics for Different Market Scenarios in Abides

Tableaux for Policy Synthesis for MDPs with PCTL* Constraints

Stability-Constrained Markov Decision Processes Using MPC

Related Research

Metrics for Markov Decision Processes with Infinite State Spaces

Self-Triggered Markov Decision Processes

Bridging the gap between QP-based and MPC-based RL

Safe Reinforcement Learning with Stability Safety Guarantees Using Robust MPC

Bayesian regularization of empirical MDPs

Similarity metrics for Different Market Scenarios in Abides

Tableaux for Policy Synthesis for MDPs with PCTL* Constraints