Joint Differentiable Optimization and Verification for Certified Reinforcement Learning

by   Yixuan Wang, et al.

In model-based reinforcement learning for safety-critical control systems, it is important to formally certify system properties (e.g., safety, stability) under the learned controller. However, as existing methods typically apply formal verification after the controller has been learned, it is sometimes difficult to obtain any certificate, even after many iterations between learning and verification. To address this challenge, we propose a framework that jointly conducts reinforcement learning and formal verification by formulating and solving a novel bilevel optimization problem, which is differentiable by the gradients from the value function and certificates. Experiments on a variety of examples demonstrate the significant advantages of our framework over the model-based stochastic value gradient (SVG) method and the model-free proximal policy optimization (PPO) method in finding feasible controllers with barrier functions and Lyapunov functions that ensure system safety and stability.


Safety-aware Adaptive Reinforcement Learning with Applications to Brushbot Navigation

This paper presents a safety-aware learning framework that employs an ad...

Proximal Reliability Optimization for Reinforcement Learning

Despite the numerous advances, reinforcement learning remains away from ...

Control of Stochastic Quantum Dynamics with Differentiable Programming

Controlling stochastic dynamics of a quantum system is an indispensable ...

Scalable Synthesis of Verified Controllers in Deep Reinforcement Learning

There has been significant recent interest in devising verification tech...

Reinforcement Learning for Robotics and Control with Active Uncertainty Reduction

Model-free reinforcement learning based methods such as Proximal Policy ...

Episodic Learning with Control Lyapunov Functions for Uncertain Robotic Systems

Many modern nonlinear control methods aim to endow systems with guarante...

Please sign up or login with your details

Forgot password? Click here to reset