CARL: Conditional-value-at-risk Adversarial Reinforcement Learning

09/20/2021

∙

by M. Godbout, et al.

∙

Université Laval

∙

Montréal Institute of Learning Algorithms

∙

In this paper we present a risk-averse reinforcement learning (RL) method called Conditional value-at-risk Adversarial Reinforcement Learning (CARL). To the best of our knowledge, CARL is the first game formulation for Conditional Value-at-Risk (CVaR) RL. The game takes place between a policy player and an adversary that perturbs the policy player's state transitions given a finite budget. We prove that, at the maximin equilibrium point, the learned policy is CVaR optimal with a risk tolerance explicitly related to the adversary's budget. We provide a gradient-based training procedure to solve CARL by formulating it as a zero-sum Stackelberg Game, enabling the use of deep reinforcement learning architectures and training algorithms. Finally, we show that solving the CARL game does lead to risk-averse behaviour in a toy grid environment, also confirming that an increased adversary produces increasingly cautious policies.

READ FULL TEXT VIEW PDF

References

[1] P. Artzner, F. Delbaen, J. Eber, and D. Heath (1999) Coherent measures of risk. Mathematical finance 9 (3), pp. 203–228. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[2] M. G. Bellemare, W. Dabney, and R. Munos (2017) A distributional perspective on reinforcement learning. In
International Conference on Machine Learning
, pp. 449–458. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[3] M. Chevalier-Boisvert, L. Willems, and S. Pal (2018) Minimalistic gridworld environment for openai gym. GitHub. Note: https://github.com/maximecb/gym-minigrid Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[4] Y. Chow, A. Tamar, S. Mannor, and M. Pavone (2015) Risk-sensitive and robust decision-making: a cvar optimization approach. In NIPS, pp. 1522–1530. External Links: Link Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning, CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[5] T. Fiez, B. Chasnov, and L. Ratliff (2020-13–18 Jul) Implicit learning dynamics in stackelberg games: equilibria characterization, convergence analysis, and empirical study. In Proceedings of the 37th International Conference on Machine Learning, H. D. III and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, pp. 3133–3144. External Links: Link Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[6] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[7] R. Keramati, C. Dann, A. Tamkin, and E. Brunskill (2020) Being optimistic to be conservative: quickly learning a cvar policy. In
Proceedings of the AAAI Conference on Artificial Intelligence
, Vol. 34, pp. 4436–4443. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[8] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, et al. (2011) Towards fully autonomous driving: systems and algorithms. In 2011 IEEE intelligent vehicles symposium (IV), pp. 163–168. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[9] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein (2017)
Unrolled generative adversarial networks
. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, External Links: Link Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[10] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta (2017) Robust adversarial reinforcement learning. In International Conference on Machine Learning, pp. 2817–2826. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[11] A. Rajeswaran, I. Mordatch, and V. Kumar (2020-13–18 Jul) A game theoretic framework for model based reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, H. D. III and A. Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, pp. 7953–7963. External Links: Link Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[12] F. Schubert, T. Eimer, B. Rosenhahn, and M. Lindauer (2021) Automatic risk adaptation in distributional reinforcement learning. External Links: 2106.06317 Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[13] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. (2018) A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362 (6419), pp. 1140–1144. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[14] R. S. Sutton, A. G. Barto, et al. (1998) Introduction to reinforcement learning. Vol. 135, MIT press Cambridge. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[15] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. (2019) Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature 575 (7782), pp. 350–354. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[16] H. Von Stackelberg (2010) Market structure and equilibrium. Springer Science & Business Media. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.
[17] J. Wiens, S. Saria, M. Sendak, M. Ghassemi, V. X. Liu, F. Doshi-Velez, K. Jung, K. Heller, D. Kale, M. Saeed, et al. (2019) Do no harm: a roadmap for responsible machine learning for health care. Nature medicine 25 (9), pp. 1337–1340. Cited by: CARL: Conditional-value-at-risk Adversarial Reinforcement Learning.

CARL: Conditional-value-at-risk Adversarial Reinforcement Learning

Authors

Risk Averse Robust Adversarial Reinforcement Learning

Adaptive Sampling for Stochastic Risk-Averse Learning

Robust Market Making via Adversarial Reinforcement Learning

Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning

Robust Reinforcement Learning using Adversarial Populations

Robust Risk-Aware Reinforcement Learning

Deep Reinforcement Learning of Transition States

References

CARL: Conditional-value-at-risk Adversarial Reinforcement Learning

Authors

Related Research

Risk Averse Robust Adversarial Reinforcement Learning

Adaptive Sampling for Stochastic Risk-Averse Learning

Robust Market Making via Adversarial Reinforcement Learning

Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning

Robust Reinforcement Learning using Adversarial Populations

Robust Risk-Aware Reinforcement Learning

Deep Reinforcement Learning of Transition States

This week in AI

References