CARL: Conditional-value-at-risk Adversarial Reinforcement Learning

09/20/2021
by   M. Godbout, et al.
0

In this paper we present a risk-averse reinforcement learning (RL) method called Conditional value-at-risk Adversarial Reinforcement Learning (CARL). To the best of our knowledge, CARL is the first game formulation for Conditional Value-at-Risk (CVaR) RL. The game takes place between a policy player and an adversary that perturbs the policy player's state transitions given a finite budget. We prove that, at the maximin equilibrium point, the learned policy is CVaR optimal with a risk tolerance explicitly related to the adversary's budget. We provide a gradient-based training procedure to solve CARL by formulating it as a zero-sum Stackelberg Game, enabling the use of deep reinforcement learning architectures and training algorithms. Finally, we show that solving the CARL game does lead to risk-averse behaviour in a toy grid environment, also confirming that an increased adversary produces increasingly cautious policies.

READ FULL TEXT
research
04/26/2022

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to find near-optimal policies f...
research
10/28/2019

Adaptive Sampling for Stochastic Risk-Averse Learning

We consider the problem of training machine learning models in a risk-av...
research
03/31/2019

Risk Averse Robust Adversarial Reinforcement Learning

Deep reinforcement learning has recently made significant progress in so...
research
02/27/2022

Neural-Progressive Hedging: Enforcing Constraints in Reinforcement Learning with Stochastic Programming

We propose a framework, called neural-progressive hedging (NP), that lev...
research
11/20/2019

Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning

Large-scale screening for potential threats with limited resources and c...
research
08/23/2021

Robust Risk-Aware Reinforcement Learning

We present a reinforcement learning (RL) approach for robust optimisatio...
research
03/03/2020

Robust Market Making via Adversarial Reinforcement Learning

We show that adversarial reinforcement learning (ARL) can be used to pro...

Please sign up or login with your details

Forgot password? Click here to reset