Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments

by   Yixuan Wang, et al.

It is quite challenging to ensure the safety of reinforcement learning (RL) agents in an unknown and stochastic environment under hard constraints that require the system state not to reach certain specified unsafe regions. Many popular safe RL methods such as those based on the Constrained Markov Decision Process (CMDP) paradigm formulate safety violations in a cost function and try to constrain the expectation of cumulative cost under a threshold. However, it is often difficult to effectively capture and enforce hard reachability-based safety constraints indirectly with such constraints on safety violation costs. In this work, we leverage the notion of barrier function to explicitly encode the hard safety constraints, and given that the environment is unknown, relax them to our design of generative-model-based soft barrier functions. Based on such soft barriers, we propose a safe RL approach that can jointly learn the environment and optimize the control policy, while effectively avoiding unsafe regions with safety probability optimization. Experiments on a set of examples demonstrate that our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.


page 1

page 2

page 3

page 4


Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees

We consider the problem of learning an episodic safe control policy that...

Lyapunov Barrier Policy Optimization

Deploying Reinforcement Learning (RL) agents in the real-world require t...

AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning

Safety is a critical hurdle that limits the application of deep reinforc...

Constrained Markov Decision Processes via Backward Value Functions

Although Reinforcement Learning (RL) algorithms have found tremendous su...

Learning Safety Constraints from Demonstrations with Unknown Rewards

We propose Convex Constraint Learning for Reinforcement Learning (CoCoRL...

Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations

Training-time safety violations have been a major concern when we deploy...

Stranding Risk for Underactuated Vessels in Complex Ocean Currents: Analysis and Controllers

Low-propulsion vessels can take advantage of powerful ocean currents to ...

Please sign up or login with your details

Forgot password? Click here to reset