Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

08/26/2021
by   Baiyu Peng, et al.
0

Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constrained RL methods like the penalty methods and the Lagrangian methods either exhibit periodic oscillations or learn an over-conservative or unsafe policy. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. We first review the constrained policy optimization process from a feedback control perspective, which regards the penalty weight as the control input and the safe probability as the control output. Based on this, the penalty method is formulated as a proportional controller, and the Lagrangian method is formulated as an integral controller. We then unify them and present a proportional-integral Lagrangian method to get both their merits, with an integral separation technique to limit the integral value in a reasonable range. To accelerate training, the gradient of safe probability is computed in a model-based manner. We demonstrate our method can reduce the oscillations and conservatism of RL policy in a car-following simulation. To prove its practicality, we also apply our method to a real-world mobile robot navigation task, where our robot successfully avoids a moving obstacle with highly uncertain or even aggressive behaviors.

READ FULL TEXT

page 1

page 9

page 10

research
02/17/2021

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

Safety is essential for reinforcement learning (RL) applied in real-worl...
research
07/08/2020

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Lagrangian methods are widely used algorithms for constrained optimizati...
research
12/19/2020

Model-Based Actor-Critic with Chance Constraint for Stochastic System

Safety constraints are essential for reinforcement learning (RL) applied...
research
11/25/2021

Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning

In the trial-and-error mechanism of reinforcement learning (RL), a notor...
research
12/14/2021

Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

Reinforcement Learning (RL) agents in the real world must satisfy safety...
research
03/02/2022

Chance-Constrained Iterative Linear-Quadratic Stochastic Games

Dynamic game arises as a powerful paradigm for multi-robot planning, for...
research
12/29/2018

SPI-Optimizer: an integral-Separated PI Controller for Stochastic Optimization

To overcome the oscillation problem in the classical momentum-based opti...

Please sign up or login with your details

Forgot password? Click here to reset