Safe Reinforcement Learning of Control-Affine Systems with Vertex Networks

03/20/2020
by   Liyuan Zheng, et al.
5

This paper focuses on finding reinforcement learning policies for control systems with hard state and action constraints. Despite its success in many domains, reinforcement learning is challenging to apply to problems with hard constraints, especially if both the state variables and actions are constrained. Previous works seeking to ensure constraint satisfaction, or safety, have focused on adding a projection step to a learned policy. Yet, this approach requires solving an optimization problem at every policy execution step, which can lead to significant computational costs. To tackle this problem, this paper proposes a new approach, termed Vertex Networks (VNs), with guarantees on safety during exploration and on learned control policies by incorporating the safety constraints into the policy network architecture. Leveraging the geometric property that all points within a convex set can be represented as the convex combination of its vertices, the proposed algorithm first learns the convex combination weights and then uses these weights along with the pre-calculated vertices to output an action. The output action is guaranteed to be safe by construction. Numerical examples illustrate that the proposed VN algorithm outperforms vanilla reinforcement learning in a variety of benchmark control tasks.

READ FULL TEXT
research
07/17/2022

Robust Action Governor for Uncertain Piecewise Affine Systems with Non-convex Constraints and Safe Reinforcement Learning

The action governor is an add-on scheme to a nominal control loop that m...
research
02/24/2021

Towards Safe Continuing Task Reinforcement Learning

Safety is a critical feature of controller design for physical systems. ...
research
05/16/2022

Reachability Constrained Reinforcement Learning

Constrained reinforcement learning (CRL) has gained significant interest...
research
01/19/2019

Towards Physically Safe Reinforcement Learning under Supervision

This paper addresses the question of how a previously available control ...
research
10/11/2020

Safe Reinforcement Learning with Natural Language Constraints

In this paper, we tackle the problem of learning control policies for ta...
research
06/20/2020

Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies

We consider the problem of reinforcement learning when provided with a b...
research
07/02/2020

Verifiably Safe Exploration for End-to-End Reinforcement Learning

Deploying deep reinforcement learning in safety-critical settings requir...

Please sign up or login with your details

Forgot password? Click here to reset