Robust Adversarial Reinforcement Learning

03/08/2017
by   Lerrel Pinto, et al.
0

Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e.g., due to different friction or object masses). Inspired from H-infinity control methods, we note that both modeling errors and differences in training and test scenarios can be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced -- that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper and Walker2d) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.

READ FULL TEXT

page 2

page 8

03/08/2020

Deep Adversarial Reinforcement Learning for Object Disentangling

Deep learning in combination with improved training techniques and high ...
07/01/2020

Falsification-Based Robust Adversarial Reinforcement Learning

Reinforcement learning (RL) has achieved tremendous progress in solving ...
10/21/2019

Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer

Learning robotic control policies in the real world gives rise to challe...
11/13/2022

Goal-Conditioned Reinforcement Learning in the Presence of an Adversary

Reinforcement learning has seen increasing applications in real-world co...
12/01/2019

Adversary A3C for Robust Reinforcement Learning

Asynchronous Advantage Actor Critic (A3C) is an effective Reinforcement ...
01/26/2019

Action Robust Reinforcement Learning and Applications in Continuous Control

A policy is said to be robust if it maximizes the reward while consideri...
06/24/2019

Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals

This paper studies reinforcement learning (RL) under malicious falsifica...

Code Repositories

RARL

Tensorflow implementation for Robust Adversarial Reinforcement Learning: https://arxiv.org/pdf/1703.02702.pdf


view repo

rllab-adv

Code to train RL agents along with Adversarial distrubance agents https://arxiv.org/abs/1703.02702


view repo

Please sign up or login with your details

Forgot password? Click here to reset