Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

1 Introduction

In reinforcement learning (RL), agents learn to act by trial and error in an unknown environment. The majority of RL algorithms allow agents to freely explore the environment and exploit any actions that might improve the reward. However, actions that lead to high rewards usually come with high risks. In a safety-critical environment, it is important to enforce safety in the RL algorithm, and a natural way to enforce safety is to incorporate constraints. A standard formulation for RL with safety constraints is the constrained Markov Decision Process (CMDP) framework CMDP , where the agents need to maximize the long-term reward while satisfying the constraints on the long-term cost. Applications of CMDPs include windmill control wind where we need to maximize the average reward (e.g., generated power) while bounding the long-term wear-and-tear cost on critical components (e.g., wind turbine). Another important example is communication network control where we need to maximize network utility while bounding the long-term arrival rate below the long-term service rate in order to maintain network stability (Chapter 1.1 in CMDP ).

While optimal policies for finite CMDPs with known models can be obtained by linear programming

LP1

, it cannot scale to high-dimensional continuous control tasks due to curse of dimensionality. Recently, there have been RL algorithms that work for high-dimensional CMDPs based on advances in policy search algorithms

TRPO ; A3C . In particular, two constrained policy search algorithms enjoy state-of-the-art performance for CMDPs: Primal-Dual Optimization (PDO) PDO and Constrained Policy Optimization (CPO) CPO . PDO is based on Lagrangian relaxation and updates parameters in primal and dual spaces in turn. Specifically, the primal policy update uses the policy gradient descent while the dual variable update uses the dual gradient ascent. By comparison, CPO differs from PDO in the dual update procedure, where the dual variable is obtained from scratch by solving a carefully-designed optimization problem in each iteration, in order to enforce safety constraints throughout training. Besides PDO and CPO, there exist other methods for solving CMDPs uchibe2007constrained ; ammar2015safe ; held2017probabilistically , but these approaches are usually computationally intensive or only apply to some specific CMDP models and domains.

A notable feature of existing constrained policy search approaches (e.g., PDO and CPO) is that they only use on-policy samples¹¹1On-policy samples refer to those generated by the currently-used policy while off-policy samples are generated by other unknown policies., which ensures that the information used for dual updates is unbiased and leads to stable performance improvement. However, such an on-policy dual update is sample-inefficient since historical samples are discarded. Moreover, due to the on-policy nature, dual updates are incremental and suffer from slow convergence since a (potentially large) batch of on-policy samples have to be obtained before a dual update can be made.

In this paper, we propose a policy search method for CMDPs called Accelerated Primal-Dual Optimization (APDO), which incorporates an off-policy trained dual variable in the dual update procedure while updating the policy in primal space with on-policy likelihood ratio gradient. Specifically, APDO is similar to PDO except that we perform a one-time adjustment for the dual variable with a nearly optimal dual variable trained with off-policy data after a certain number of iterations. Such a one-time adjustment process incurs negligible amortized overhead in the long term but greatly improves the sample efficiency and the convergence rate over exisiting methods. We demonstrate the effectiveness of APDO on a simulated robot locomotion task where the agent must satisfy constraints motivated by safety. The experimental results show that APDO achieves better sample efficiency and faster convergence than state-of-the-art approaches for CMDPs (e.g., PDO and CPO).

Another line of work considers merging the on-policy and off-policy policy gradient updates to improve sample efficiency. Examples of these approaches include Q-Prop Q-prop , IPG IPG , etc. These approaches are designed for unconstrained MDPs and can be applied to the primal policy update. In contrast, APDO leverages off-policy samples for dual updates and is complementary to these efforts on merging on-policy and off-policy policy gradients.

2 Constrained Markov Decision Process

A Markov Decision Process (MDP) is represented by a tuple, $(S, A, R, P, p_{0})$ , where $S$ is the set of states, $A$ is the set of actions, $R : S \times A \times S \mapsto R$ is the reward function, $P : S \times A \times S \mapsto [0, 1]$

is the transition probability function (where

P (s^{'} | s, a)

is the transition probability from state

s

to state

s^{'}

given action

a

), and

p_{0} : S \mapsto [0, 1]

is the initial state distribution. A stationary policy

π : S \mapsto P (A)

corresponds to a mapping from states to a probability distribution over actions. Specifically,

π (a | s)

is the probability of selecting action

a

in state

s

. The set of all stationary policies is denoted by

Π

. In this paper, we search policy within a parametrized stationary policy class

Π_{θ} \subset Π

(e.g., a neural network policy class with weight

θ

). We may write a policy

π

π (θ)

to emphasize its dependence on the parameter

θ

. The long-term discounted reward under policy

π

is denoted as

R (π) = E_{τ \sim π} [\sum_{t = 0}^{\infty} γ^{t} R (s_{t}, a_{t}, s_{t + 1})]

, where

γ \in [0, 1)

is the discount factor,

τ = (s_{0}, a_{0}, s_{1}, a_{1}, \dots)

denotes a trajectory, and

τ \sim π

means that the distribution over trajectories is determined by policy

π

, i.e.,

s_{0} \sim p_{0}, a_{t} \sim π (\cdot | s_{t}), s_{t + 1 \sim P (\cdot | s_{t}, a_{t})}

A constrained Markov Decision Process (CDMP) is an MDP augmented with constraints on long-term discounted costs. Specifically, we augment the ordinary MDP with $m$ cost functions $C_{1}, \dots, C_{m}$ , where each cost function $C_{i} : S \times A \times S \mapsto R$ is a mapping from transition tuples to costs. The long-term discounted cost under policy $π$ is similarly defined as $C_{i} (π) = E_{τ \sim π} [\sum_{t = 0}^{\infty} γ^{t} C_{i} (s_{t}, a_{t}, s_{t + 1})]$ , and the corresponding limit is $d_{i}$ . In CMDP, we aim to select a policy $π$ that maximizes the long-term reward $R (π)$ while satisfying the constraints on the long-term costs $C_{i} (π) \leq d_{i}, \forall i \in [m]$ , i.e.,

\begin{matrix} π^{*} = & arg max π \in Π_{θ} R (π) s.t. C_{i} (π) \leq d_{i}, \forall i \in [m] . \end{matrix}

(1)

3 Algorithm

To solve CMDPs, we employ the Lagrangian relaxation procedure (Chapster 3 in lagrange ). Specifically, the Lagrangian function for the CMDP problem (1) is

(2)

where $λ = (λ_{1}, \dots, λ_{m})$ is the Lagrangian multiplier. Then the constrained problem (1) can be converted to the following unconstrained problem:

(π^{*}, λ^{*}) = arg min λ \geq 0 max π \in Π_{θ} L (π, λ) .

(3)

To solve the unconstrained minimax problem (3), a canonical approach is to use the iterative primal-dual method where in each iteration we update the primal policy $π$ and the dual variable $λ$ in turn. The primal-dual update procedures at iteration $k$ are as follows:

$∙$ Fix $λ = λ^{(k)}$ and perform policy gradient update: $θ_{k + 1} = θ_{k} + α_{k} \nabla_{θ} (L (π (θ), λ^{(k)})) |_{θ = θ_{k}},$ where $α_{k}$ is the step size. The policy gradient could be on-policy likelihood ratio policy gradient (e.g., REINFORCE REINFORCE and TRPO TRPO ) or off-policy deterministic policy gradient (e.g., DDPG DDPG ).

$∙$ Fix $π = π_{k}$ and perform dual update $λ^{(k + 1)} = f_{k} (λ^{(k)}, π_{k})$ . Existing methods for CMDPs, such as PDO and CPO, differ in the choice of the dual update procedure $f_{k} (\cdot)$ . For example, PDO uses the simple dual gradient ascent $λ_{i}^{(k + 1)} = [λ_{i}^{(k)} + β_{k} (C_{i} (π_{k}) - d_{i})]^{+},$ where $β_{k}$ is the step size and $[x]^{+} = max {0, x}$ is the projection onto the dual space $λ \geq 0$ . By comparison, CPO derives the dual variable $λ^{(k + 1)}$ by solving an optimization problem from scratch in order to enforce the constraints in every iteration.

However, the dual update procedures used in existing methods (e.g., PDO and CPO) are incremental and only use on-policy samples, resulting sample inefficiency and slow convergence to the optimal primal-dual solution $(λ^{*}, π^{*})$ . In this paper, we propose to incorporate an off-policy trained dual variable in the dual update procedure in order to improve sample efficiency and speed up the search for the optimal dual variable $λ^{*}$ . The algorithm is called Accelerated Primal-Dual Optimization (APDO) and is described in Algorithm 1. APDO is similar to PDO where in most iterations the dual variable is updated according to the simple dual gradient ascent (step 6), but the key innovation of APDO is that there is a one-time dual adjustment with an off-policy trained dual variable $λ^{OFF}$ after $K^{adj}$ iterations (steps 7-10). The off-policy trained $λ^{OFF}$ is obtained by running an off-policy algorithm for CMDPs with the historical data stored in the replay buffer. We provide a primal-dual version of the DDPG algorithm in the supplementary material for training $λ^{OFF}$ . Although the off-policy trained dual variable $λ^{OFF}$ could be biased, it provides a nearly optimal point for further fine tuning of the dual variable using new on-policy data.

The improvement of sample efficiency in APDO is due to the fact that off-policy training can repeatedly exploit historical data while on-policy update only uses each sample once; the acceleration effect of APDO is due to the fact that off-policy training directly solves for the optimal dual variable offline, thus avoiding the slow on-policy dual update as in the existing approaches where only one dual update can be taken after a large batch of samples are obtained.

Note that the adjustment epoch

K^{adj}

is an important parameter in APDO. Using a small

K^{adj}

avoids slow incremental dual update early, but the dual estimate

λ^{OFF}

could be highly biased and inaccurate due to insufficient amount of data. On the other hand, using a larger

K^{adj}

provides a more accurate dual estimate at the expense of delayed adjustment.

1: Initialize policy

π_{0} \in Π_{θ}

, replay buffer

R = \emptyset

2: for

k = 0, 1, 2 \dots,

3: Sample a set of trajectories

D_{k}

under the current policy

π_{k} = π (θ_{k})

(containing

T

samples)

4: Add the sampled data

{(s_{t}, a_{t}, r_{t}, c_{t}, s_{t + 1})}_{t = 0}^{T}

to the replay buffer

R

5: Update the primal policy with any on-policy likelihood ratio gradient method (e.g., TRPO) using the sampled on-policy trajectories

D_{k}

and the current dual variable

λ^{(k)}

6: Update the dual variable with dual gradient ascent:

λ_{i}^{(k + 1)} = [λ_{i}^{(k)} + β_{k} (C_{i} (π_{k}) - d_{i})]^{+}, \forall i

7: if

k = K^{adj}

then

8: Compute the off-policy trained

λ^{OFF}

with the replay buffer

R

(e.g., using the primal-dual DDPG in the supplementary material)

9: Set

λ^{(k + 1)} = λ^{OFF}

10: end if

11: end for

Algorithm 1 Accelerated Primal-Dual Policy Optimization (APDO)

4 Experiments

We evaluate APDO against two state-of-the-art algorithms for solving CMDPs (i.e., CPO and PDO) on a simple point-gather control task in MuJoCo mujoco with an additional safety constraint as used in CPO . All experiments are implemented in rllab rllab . The detailed task description and experiment parameters are provided in the supplementary material. In particular, for APDO we set the adjustment epoch $K^{adj} = 5$ , and additional experimental results regarding the effect of $K^{adj}$ are also given in the supplementary material.

Figure 1 shows the learning curves for APDO, CPO and PDO under cost constraints. It can be observed from Fugure 1(b) that APDO enforced constraints successfully to the limit value as approximately same speed as CPO did. More importantly, APDO generally outperforms CPO on reward performance without compromising constraint stabilization, thus achieving better sample efficiency. For example, CPO takes 90 epochs to achieve an average reward of 11 while satisfying the safety constraint. By comparison, APDO only takes 45 epochs to achieve the same point, which corresponds to 2x improvement in sample efficiency over CPO in this task. In addition, PDO fails to enforce the safety constraint during the first 150 epochs due to its slow convergence. Using a larger step size may help speed up the convergence but in this case PDO will over-correct in response to constraint violations and behave too conservatively. We provide additional discussions on the choice of stepsize for PDO and APDO in the supplementary material.

Figure 1(c) illustrates the learning trajectory of the dual variable under PDO and APDO (note that the dual variable for CPO is not illustrated since CPO has a sophisticated recovery scheme to enforce constraints, where the dual variable may not be easily obtained). We find that APDO converges to the optimal dual variable $λ^{*}$ significantly faster than PDO. In particular, there is a “jump" of the dual variable after several epochs in APDO, due to the dual adjustment with the off-policy trained $λ^{OFF}$ . By comparison, PDO has to adjust its dual variable incrementally with on-policy data.

5 Future Work

Since the adjustment epoch is an important parameter in APDO, one important future work is to provide theoretical guidance on the setting of $K^{adj}$ . It is also very interesting (yet challenging) to provide theoretical justifications about the acceleration effects of APDO. Moreover, as we observed in the experiments, the training trajectory generated by APDO strives for the best tradeoff between improving rewards and enforcing cost constraints. One future work is to incorporate a safety parameter that controls the degree of safety awareness. By tuning the parameter, the RL algorithm should be able to make both risk-averse actions (which enforce safety constraints as soon as possible) and risk-neutral actions (which gives priority to improving rewards).

Acknowledgment

This work was supported by NSF Grant CNS-1524317 and by DARPA I2O and Raytheon BBN Technologies under Contract No. HROO l l-l 5-C-0097. The authors would also like to acknowledge Chengtao Li who provided valuable feedback on this work.

References

(1) S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi-agent, reinforcement learning for autonomous driving,” arXiv preprint arXiv:1610.03295, 2016.
(2) E. Altman, Constrained Markov decision processes. CRC Press, 1999, vol. 7.
(3) P. Tavner, J. Xiang, and F. Spinato, “Reliability analysis for wind turbines,” Wind Energy, vol. 10, no. 1, pp. 1–18, 2007.
(4) E. A. Feinberg and A. Shwartz, “Constrained dynamic programming with two discount factors: Applications and an algorithm,” IEEE Transactions on Automatic Control, vol. 44, no. 3, pp. 628–631, 1999.
(5) J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in Proceedings of the 32nd International Conference on Machine Learning (ICML-15)
, 2015, pp. 1889–1897.
(6) V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in International Conference on Machine Learning, 2016, pp. 1928–1937.
(7) Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone, “Risk-constrained reinforcement learning with percentile risk criteria,” arXiv preprint arXiv:1512.01629, 2015.
(8) J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in Proceedings of the 34nd International Conference on Machine Learning (ICML-17), 2017.
(9) E. Uchibe and K. Doya, “Constrained reinforcement learning from intrinsic and extrinsic rewards,” in Development and Learning, 2007. ICDL 2007. IEEE 6th International Conference on. IEEE, 2007, pp. 163–168.
(10) H. B. Ammar, R. Tutunov, and E. Eaton, “Safe policy search for lifelong reinforcement learning with sublinear regret,” in Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 2015, pp. 2361–2369.
(11) D. Held, Z. McCarthy, M. Zhang, F. Shentu, and P. Abbeel, “Probabilistically safe policy transfer,” arXiv preprint arXiv:1705.05394, 2017.
(12) S. Gu, T. Lillicrap, Z. Ghahramani, R. E. Turner, and S. Levine, “Q-prop: Sample-efficient policy gradient with an off-policy critic,” International Conference on Learning Representations (ICLR-17), 2017.
(13) S. Gu, T. Lillicrap, Z. Ghahramani, R. E. Turner, B. Schölkopf, and S. Levine, “Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning,” arXiv preprint arXiv:1706.00387, 2017.
(14) D. P. Bertsekas, Nonlinear programming. Athena scientific Belmont, 1999.
(15) R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3-4, pp. 229–256, 1992.
(16) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” International Conference on Learning Representations (ICLR-16), 2016.
(17) E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012, pp. 5026–5033.
(18) Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” in International Conference on Machine Learning, 2016, pp. 1329–1338.
(19) J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438, 2015.
(20) D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

Supplementary Materials

Appendix A Primal-Dual DDPG for CMDPs

In this appendix, we provide a primal-dual version of the DDPG algorithm for solving CMDPs. The primal policy update and the dual variable update in this algorithm only use the off-policy data stored in the replay buffer, which can be used to fit $λ^{OFF}$ for our APDO algorithm. For simplicity, we only present the algorithm for CMDPs with a single constraint, and the multiple-constraint case can be easily obtained. In the primal-dual DDPG algorithm, we have the following neural networks.

Reward critic Q-network $Q_{R} (s, a | θ_{R}^{Q})$ and reward target Q-network $Q_{R}^{'} (s, a | θ_{R}^{Q^{'}})$
Cost critic Q-network $Q_{C} (s, a | θ_{C}^{Q})$ and cost target Q-network $Q_{C}^{'} (s, a | θ_{C}^{Q^{'}})$
Actor policy network $μ (s | θ^{μ})$ and actor target Q-network $μ^{'} (s | θ^{μ^{'}})$

The target networks are used to slowly track the learned networks.

1: Randomly initialize reward critic Q-network

Q_{R} (s, a | θ_{R}^{Q})

, cost critic Q-network

Q_{C} (s, a | θ_{C}^{Q})

and actor network

μ (s | θ^{μ})

2: Initialize target networks:

θ_{R}^{Q^{'}} \leftarrow θ_{R}^{Q}

θ_{C}^{Q^{'}} \leftarrow θ_{C}^{Q}

θ^{μ^{'}} \leftarrow θ^{μ}

3: Initialize replay buffer

R

and dual variable

λ

4: for episode

k = 0, 1, \dots,

5: Initialize a random process

N

for action exploration

6: Receive initial state

s_{0} \sim p_{0}

7: for

t = 1, \dots, T

8: Select action

a_{t} = μ (s_{t} | θ^{μ}) + N_{t}

9: Execute action

a_{t}

and observe

r_{t}, c_{t}, s_{t + 1}

10: Store transition

(s_{t}, a_{t}, r_{t}, c_{t}, s_{t + 1})

in the replay buffer

R

11: Sample a random batch of

N

transitions

{(s_{i}, a_{i}, r_{i}, c_{i}, s_{i + 1})}_{i = 1}^{N}

from the replay buffer

R

12: Set

y_{i} = r_{i} + γ Q_{R}^{'} (s_{i + 1}, μ^{'} (s_{i + 1} | θ^{μ^{'}}) | θ_{R}^{Q^{'}}))

z_{i} = c_{i} + γ Q_{C}^{'} (s_{i + 1}, μ^{'} (s_{i + 1} | θ^{μ^{'}}) | θ_{C}^{Q^{'}}))

13: Update reward critic by minimizing

L_{R} = \frac{1}{N} \sum_{i} (y_{i} - Q_{R} (s_{i}, a_{i} | θ_{R}^{Q}))^{2}

and update cost critic by minimizing

L_{C} = \frac{1}{N} \sum_{i} (z_{i} - Q_{C} (s_{i}, a_{i} | θ_{C}^{Q}))^{2}

14: Update the actor policy using the sampled policy gradient

\nabla_{θ^{μ}} L (θ^{μ}, λ) = \frac{1}{N} \sum i \nabla_{θ^{μ}} (Q_{R} (s, μ (s | θ^{μ}) | θ_{R}^{Q}) - λ Q_{C} (s, μ (s | θ^{μ}) | θ_{C}^{Q})) {∣ ∣}_{s = s_{i}}

15: Update dual variable using the sampled dual gradient

\nabla_{λ} L (θ^{μ}, λ) = \frac{1}{N} \sum i [Q_{C} (s_{i}, μ (s_{i} | θ^{μ})) - d]

16: Update target networks:

\begin{matrix} θ_{R}^{Q^{'}} \leftarrow τ θ_{R}^{Q} + (1 - τ) θ_{R}^{Q^{'}} θ_{C}^{Q^{'}} \leftarrow τ θ_{C}^{Q} + (1 - τ) θ_{C}^{Q^{'}} θ^{μ^{'}} \leftarrow τ θ^{μ} + (1 - τ) θ^{μ^{'}} \end{matrix}

17: end for

18: end for

Algorithm 2 Primal-Dual DDPG

Appendix B Experiment Details

Task description. Specifically, a point mass receives a reward of 10 for collecting an apple, and a cost of 1 for collecting a bomb. The agent is constrained to incur no more than 0.2 cost in the long term. Two apples and eight bombs spawn on the map at the start of each episode.

Parameters for primal policy update. For all experiments, we use neural network policies with two hidden layers of sizes $(64, 32)$ with tanh non-linearity, and all of the schemes (PDO, CPO, APDO) use TRPO to update the primal policy, with a batch size 50000 and a KL-divergence step size of 0.01. The discount factor is 0.995 and the rollout length is 15. We use GAE- $λ$ GAE for estimating the regular advantages with $λ^{GAE} = 0.95$ .

Parameters for dual variable update. As for dual updates, PDO and APDO both use dual gradient ascent. Note that the step size for dual gradient ascent is important in PDO: if it is set to be too small, the dual variable won’t update quickly enough to meaningfully enforce the constraint; if it is too high, the algorithm will over-correct in response to constraint violations and behave too conservatively CPO As a result, picking a proper step size is critical and difficult in PDO. We experiment with different step sizes ${0.01, 0.05, 0.1, 0.5}$ and find that 0.1 works best for PDO, and the reported results of PDO are also under the step size 0.1. By comparison, selecting step size in APDO is much easier since the one-time off-policy dual adjustment directly boosts the dual variable to a "nearly optimal" point and we only need to choose a relatively small step size in order to do fine-tuning after the adjustment. For the reported experimental results, we also set the step size to be 0.1 for APDO for the fairness of comparison. As for CPO, we adopt the same set of parameters as in original CPO paper CPO (specially, the parameters used in the point-gather task).

Parameters for training $λ^{OFF}$ . We use primal-dual DDPG to train $λ^{OFF}$ . The reward critic network $Q_{R} (s, a | θ_{R}^{Q})$ and cost critic network $Q_{C} (s, a | θ_{C}^{Q})$ ) is parametrized by a neural network with two hidden layers of sizes $(100, 100)$ with tanh nonlinearity, respectively. The actor policy network $μ (s | θ^{μ})$ is represented by a neural network with two hidden layers of sizes $(64, 32)$ with tanh nonlinearity. The learning rates for the reward/cost critic Q-network and the actor policy network are all $10^{- 3}$ and these networks are updated with Adam adam . The update for the dual variable in primal-dual DDPG employs simple dual gradient ascent and the step size for updating the dual variable in the primal-dual DDPG is set to be $10^{- 2}$ . The mini-batch size is $N = 64$ . We also use a soft target networks with $τ = 0.001$ . The off-policy training is executed for $5 \times 10^{5}$ primal-dual iterations. Since off-policy algorithms like DDPG are usually unstable, we set $λ^{OFF}$ to be the average of all historical dual variables throughout the off-policy training trajectory. The max replay buffer size is $10^{6}$ .

Effect of $K^{adj}$ . Figure 2 shows the effect of adjustment epoch $K^{adj}$ on the performance of APDO, where we experiment with $K^{adj} \in {1, 5, 10}$ . It is observed that using a smaller $K^{adj}$ avoids slow incremental dual update earlier, but due to limited amount of available samples in the replay buffer the off-policy dual estimate $λ^{OFF}$ could be highly biased and inaccurate. On the other hand, using a larger $K^{adj}$ provides a more accurate dual estimate at the expense of delayed adjustment.

Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

A Primal Approach to Constrained Policy Optimization: Global Optimality and Finite-Time Analysis

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach

Constrained Variational Policy Optimization for Safe Reinforcement Learning

Cautious Reinforcement Learning via Distributional Risk in the Dual Domain

Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Data

State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards

1 Introduction

2 Constrained Markov Decision Process

3 Algorithm

4 Experiments

5 Future Work

Acknowledgment

References

Supplementary Materials

Appendix A Primal-Dual DDPG for CMDPs

Appendix B Experiment Details

Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

Related Research

A Primal Approach to Constrained Policy Optimization: Global Optimality and Finite-Time Analysis

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach

Constrained Variational Policy Optimization for Safe Reinforcement Learning

Cautious Reinforcement Learning via Distributional Risk in the Dual Domain

Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Data

State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards

1 Introduction

2 Constrained Markov Decision Process

3 Algorithm

4 Experiments

5 Future Work

Acknowledgment

References

Supplementary Materials

Appendix A Primal-Dual DDPG for CMDPs

Appendix B Experiment Details