Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

I Introduction

Reinforcement learning (RL) is promising for solving nonlinear optimal control problems and has received significant attention in the past decades, see [grondman2012survey, kim2021dynamic] and the references therein. Until recently, significant progress has been made on RL with the actor-critic structure for continuous control tasks [grondman2012survey, haarnoja2018soft, liu2013policy, zhang2009neural, jiang2012computational, lim2020prediction, wang2021intelligent]. In actor-critic RL, the value function and control policy are represented by the critic and actor networks, respectively, and learned via extensive policy exploration and exploitation. However, the resulting learning-based control system might not guarantee safety for systems with state and stability constraints. It is known that safety constraint satisfaction is crucial besides optimality in many real-world robot control applications [garcia2015comprehensive, chow2019lyapunov]. For instance, autonomous driving has been viewed as a promising technology that will bring fundamental changes to everyday life. Still, one of the crucial issues concerns how to learn to drive safely under dynamic and unknown environments with unexpected obstacles. For these practical reasons, many safe RL algorithms have been recently developed for safety-critical systems, see e.g. [chow2018lyapunovbased, chow2019lyapunov, berkenkamp2018safe, garcia2015comprehensive, srinivasan2020learning, yu2019convergent, xu2021crpo, ma2021model, simao2021alwayssafe, amani2021safe, yang2021accelerating, huh2020safe, zheng2021safe, marvi2021safe, chen2021context, li2021safe, brunke2021safe, richards2018lyapunov] and the references therein. Note that there are fruitful works in adaptive control with constraints, but the technique used is different from that in RL, for related references in adaptive control might refer to [sun2016neural, kong2019adaptive].

In general, current safe RL solutions can be categorized into three main approaches. (i) The first family utilizes a unique mechanism in the learning procedure for safe policy optimization using, e.g., control barrier functions [yang2019safety, ma2021model, marvi2021safe], formal verification [fulton2018safe, turchetta2020safe], shielding [alshiekh2018safe, zanon2020safe, thananjeyan2021recovery], and external intervention [saunders2017trial, wagener2021safe]. These methods are prone to a safe-biased learning by sacrificing greatly on the performance. And some of them rely on extra human interference [saunders2017trial, wagener2021safe]. (ii) The second family proposes safe RL algorithms via primal-dual methods [chen2016stochastic, paternain2019safe, xu2021crpo, ding2021provably]. In the resulting optimization problem, the Lagrangian multiplier serves as an extra weight whose update is noted sensitive to the control performance [xu2021crpo]. Moreover, some optimization problems such as those with ellipsoidal constraints (covered in this work) could not satisfy the strong duality condition [zhang2021robust]. (iii) The third is reward/cost shaping-based RL approaches [geibel2005risk, balakrishnan2019structured, hu2020learning, tessler2018reward] where the cost functions are augmented with various safety-related parts, e.g., barrier functions. As stated in [paternain2019safe], such a design only informs the goal of guaranteeing safety by minimizing the reshaped cost function but fails to guide how to achieve it well through an actor-critic structure design. Consequently, weights of actor and critic networks are prone to divergence in the training process due to the abrupt changes of the cost function caused by the safety-related terms in approaching the constraint boundary. These issues motivated our barrier function-based (simplified as barrier-based) actor-critic structure. Moreover, few works have addressed the safe RL algorithm design under time-varying safety constraints.

This work proposes a model-based safe RL algorithm with theoretical guarantees for optimal control with time-varying state and control constraints. A new barrier-based control policy (BCP) structure is constructed in the proposed safe RL approach, generating repulsive control forces as states and controls move toward the constraint boundaries. Moreover, the time-varying constraints are addressed by a multi-step policy evaluation (MPE). The closed-loop theoretical property of our approach under nominal and perturbed cases and the convergence condition of the barrier-based actor-critic (BAC) learning algorithm is derived. The effectiveness of our approach is tested on both simulations and real-world intelligent vehicles. Our contributions are summarized as follows.

We proposed a safe RL for optimal control under time-varying state and control constraints. Under certain conditions (see Sections LABEL:sec:32-A and -B), safety can be guaranteed in both online and offline learning scenarios. The performance and advantages of the proposed approach are achieved by two novel designs. The first is a barrier-based control policy to ensure safety with an actor-critic structure. The second is a multi-step evaluation mechanism to predict the control policy’s future influence on the value function under time-varying safety constraints and guide the policy to update safely.
We proved that the proposed safe RL could guarantee stability and robustness in the nominal scenario and under external disturbances, respectively. And the convergence condition of the BAC learning algorithm was derived by the Lyapunov method.
The proposed approach was applied to solve an integrated path following and collision avoidance problem of intelligent vehicles so that the control performance can be optimized with theoretical guarantees even with external disturbances. (i) Extensive simulation results illustrate that our approach outperforms other state-of-the-art safe RL methods in learning safety and performance. (ii) We verified our control policy’s offline sim-to-real transfer capability and real-world online learning performance. The experimental results reveal that our approach outperforms a standard model predictive control (MPC) algorithm in terms of safety and optimality and shows an impressive sim-to-real transfer capability and a satisfactory online control performance.

The remainder of the paper is organized as follows. Section II introduces the considered control problem and preliminary solutions. Section III presents the proposed safe RL approach and the BAC learning algorithm, while Section IV presents the main theoretical results. Section V shows the real-world experimental results, while some conclusions are drawn in Section VI. Some additional simulation results for theoretical verification are given in the appendix.

Notation: We denote $N$ and $N_{a}^{b}$ as the set of natural numbers and integers $a, a + 1, \dots, b$

. For a vector

x \in R^{n}

, we denote

∥ x ∥_{Q}^{2}

x^{⊤} Q x

and

∥ x ∥

as the Euclidean norm. For a function

f (x)

with an argument

x

, we denote

▽ f (x)

as the gradient to

x

. For a function

f (x, u)

with arguments

x

and

u

, we denote

▽_{z} f (x, u)

as the partial gradient to

z

z = x

u

. Given a matrix

A \in R^{n \times n}

, we use

λ_{m i n} (A)

(

λ_{m a x} (A)

) to denote the minimal (maximal) eigenvalues. We denote

I n t (Z)

as the interior of a general set

Z

. For variables

z_{i} \in R^{q_{i}}

i \in N_{1}^{M}

, we define

(z_{1}, z_{2}, \dots, z_{M}) = [z_{1}^{⊤} z_{2}^{⊤} \dots z_{M}^{⊤}]^{⊤} \in R^{q}

, where

q = \sum_{i = 1}^{M} q_{i}

Ii Control problem formulation and definitions

Ii-a Control problem

The considered system under control is a class of discrete-time nonlinear systems described by

\begin{matrix} x_{k + 1} = f (x_{k}, u_{k}) \end{matrix}

(1)

where $x_{k} \in X_{k} \subseteq R^{n}$ and $u_{k} \in U_{k} \subseteq R^{m}$ are the state and input variables, $k$ is the discrete-time index, $X_{k} = {x \in R^{n} | G_{x, k}^{i} (x) \leq 0, \forall i \in N_{1}^{p_{x}}}$ and $U_{k} = {u \in R^{m} | G_{u, k}^{i} (u) \leq 0, \forall i \in N_{1}^{p_{u}}}$ are convex sets that represent time-varying constraints, ${0} \subseteq U_{k}$ , $\forall k \in N$ , $W$ is a bounded compact set; functions $G_{z, k}^{i} (z) \in R$ for $z = x, u$ , are assumed to be $C^{2}$ ; $f$ is the state transition function and $f (0, 0) = 0$ .

Starting from any initial condition $x_{0} \in X_{0}$ , the control objective is to find an optimal control policy $u_{k} = u (x_{k}) \in U_{k}$ that minimizes a quadratic regulation cost function of type

J (x_{0}, u_{0 : + \infty}) = + \infty \sum k = 0 γ^{k} r (x_{k}, u_{k})

(2)

subject to model (1), $x_{k} \in X_{k}$ , and $u_{k} \in U_{k}$ , $\forall k \in N$ ; where

r (x_{k}, u_{k}) = ∥ x_{k} ∥_{Q}^{2} + ∥ u_{k} ∥_{R}^{2},

and $Q = Q^{⊤} \in R^{n \times n}$ , $R = R^{⊤} \in R^{m \times m}$ , $Q, R ≻ 0$ , $γ$ is a discount factor.

Definition 1 (Local stabilizability [zhang2021robust])

System (1) with $w = 0$ is stabilizable on $X_{k} \times U_{k}$ if, for any $x_{0} \in X_{k}$ , there exists a $C^{1}$ state-feedback policy $u (x_{k}) \in U_{k}$ , $u (0) = 0$ , such that $x_{k} \in X_{k}$ and $x_{k} \to 0$ as $k \to + \infty$ .

Without loss of generality, many waypoint tracking problems in the robot control field can be naturally formed as the prescribed regulation one, with a proper coordination transformation of the reference waypoints. More generally, it is allowed that the time-varying state constraint might not contain the origin for some $k \in N$ . Typical examples can be found in, for instance, path following of mobile robots with collision avoidance, where the potential obstacle to be avoided might occupy the reference waypoints, i.e., the origin after coordination transformation. It is still reasonable to introduce the following assumption for convergence guarantee.

Assumption 1 (State constraint)

There exists a finite number $¯ k \in N$ such that ${0} \subseteq X_{k}$ as $k \geq ¯ k$ .

Assumption 2 (Lipschitz continuous)

Model (1) is Lipschitz continuous in $X_{k} \times U_{k}$ , for all $k \in N_{1}^{\infty}$ , i.e., there exists a Lipschitz constant $0 < L_{f} < + \infty$ such that for all $x_{1}, x_{2} \in X_{k}$ and $C^{1}$ control policies with $u (x_{1}), u (x_{2}) \in U_{k}$ ,

∥ f (x_{1}, u (x_{1})) - f (x_{2}, u (x_{2})) ∥ \leq L_{f} ∥ x_{1} - x_{2} ∥ .

(3)

Assumption 3 (Model)

$∥ ▽_{u} f (x, u) ∥ \leq g_{m}$ in the domain $X_{k} \times U_{k}$ , where $g_{m} > 0$ .

Definition 2 (multi-step safe control)

For a given state $x_{k} \in X_{k}$ at time instant $k$ , a control policy $u (x_{k}) \in U_{k}$ , is $L$ -step safe for (1) if the resulting future state evolutions of (1) under $u (x_{k})$ satisfy $x_{k + i} \in X_{k + i}^{u}$ , $\forall i \in N_{1}^{L}$ , where $X_{k + i}^{u}$ is the resulting state constraint under $u (x_{k})$ .

To simplify the notation, in the rest of the paper, the super index in $X_{k}^{u}$ is neglected, i.e., we use $X_{k}$ to denote $X_{k}^{u}$ .

Ii-B Definitions on barrier functions

We also introduce the definition of barrier functions.

Definition 3 (Barrier function)

For a convex set $Z_{k} = {z \in R^{l} | G_{z, k}^{i} (z) \leq 0, \forall i \in N_{1}^{p_{z}}}$ , a barrier function is defined as

B_{k}^{o} (z) = {\begin{matrix} - \sum_{i = 1}^{p_{z}} l o g (- G_{z, k}^{i} (z)), z \in I n t (Z_{k}) + \infty o t h e r w i s e . \end{matrix}

(4)

The recentered transformation of $B_{k}^{o} (z)$ centered at $z_{c} \in R^{l}$ is defined as $B_{k}^{c} (z) = B_{k}^{o} (z) - B_{k}^{o} (z_{c}) - ▽_{z} B_{k}^{o} (z_{c})^{⊤} z$ , where $z_{c} = 0$ if ${0} \subseteq Z_{k}$ or $z_{c}$ is selected such that $z_{c} \in Z_{k}$ otherwise.

To derive a satisfactory control performance, it is suggested to select $z_{c}$ far from the set boundary of $Z_{k}$ and as the central point or its neighbor of $Z_{k}$ (if possible). It is observed that $B_{k}^{c} (z)$ satisfies

B_{k}^{c} (z_{c}) = 0 and ▽ B_{k}^{c} (z_{c}) = 0.

(5)

Lemma 1 (Relaxed barrier function [wills2004barrier])

Define a relaxed barrier function of $B_{k}^{c} (z)$ as

B_{k} (z) = {\begin{matrix} B_{k}^{c} (z) & {¯ σ}_{k} \geq κ_{b} γ_{b} (z, {¯ σ}_{k}) & {¯ σ}_{k} < κ_{b} \end{matrix}

(6)

where $κ_{b} > 0$ is a relaxing factor, ${¯ σ}_{k} = {m i n}_{i \in N_{1}^{p_{z}}} - G_{k}^{i} (z)$ , the function $γ_{b} (z, {¯ σ}_{k})$ is strictly monotone and differentiable on $(- \infty, κ_{b})$ , and $▽_{z}^{2} γ_{b} (z, {¯ σ}_{k}) \leq ▽_{z}^{2} B_{k} (z) |_{{¯ σ}_{k} = κ_{b}}$ , then there exists a matrix $H_{z_{k}} \geq ▽_{z}^{2} B_{k} (z) |_{{¯ σ}_{k} = κ_{b}},$ such that $∥ ▽_{z} B_{k} (z) ∥ \leq B_{z_{k}, m},$ $B_{z_{k}, m} = {max}_{z \in Z_{k}} ∥ 2 H_{z_{k}} (z - z_{c}) ∥$ .

Proof: For details please see [wills2004barrier]. $□$

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Authors

Model-Based Actor-Critic with Chance Constraint for Stochastic System

DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances

Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee

Lyapunov Barrier Policy Optimization

Off-policy reinforcement learning for H_∞ control design

Collision Avoidance in Tightly-Constrained Environments without Coordination: a Hierarchical Control Approach

Safe Learning of Linear Time-Invariant Systems

I Introduction

Ii Control problem formulation and definitions

Ii-a Control problem

Definition 1 (Local stabilizability [zhang2021robust])

Assumption 1 (State constraint)

Assumption 2 (Lipschitz continuous)

Assumption 3 (Model)

Definition 2 (multi-step safe control)

Ii-B Definitions on barrier functions

Definition 3 (Barrier function)

Lemma 1 (Relaxed barrier function [wills2004barrier])

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Authors

Related Research

Model-Based Actor-Critic with Chance Constraint for Stochastic System

DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances

Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee

Lyapunov Barrier Policy Optimization

Off-policy reinforcement learning for H_∞ control design

Collision Avoidance in Tightly-Constrained Environments without Coordination: a Hierarchical Control Approach

Safe Learning of Linear Time-Invariant Systems

This week in AI

I Introduction

Ii Control problem formulation and definitions

Ii-a Control problem

Definition 1 (Local stabilizability [zhang2021robust])

Assumption 1 (State constraint)

Assumption 2 (Lipschitz continuous)

Assumption 3 (Model)

Definition 2 (multi-step safe control)

Ii-B Definitions on barrier functions

Definition 3 (Barrier function)

Lemma 1 (Relaxed barrier function [wills2004barrier])