A Game-Theoretic Perspective of Generalization in Reinforcement Learning

08/07/2022
by   Chang Yang, et al.
0

Generalization in reinforcement learning (RL) is of importance for real deployment of RL algorithms. Various schemes are proposed to address the generalization issues, including transfer learning, multi-task learning and meta learning, as well as the robust and adversarial reinforcement learning. However, there is not a unified formulation of the various schemes, as well as the comprehensive comparisons of methods across different schemes. In this work, we propose a game-theoretic framework for the generalization in reinforcement learning, named GiRL, where an RL agent is trained against an adversary over a set of tasks, where the adversary can manipulate the distributions over tasks within a given threshold. With different configurations, GiRL can reduce the various schemes mentioned above. To solve GiRL, we adapt the widely-used method in game theory, policy space response oracle (PSRO) with the following three important modifications: i) we use model-agnostic meta learning (MAML) as the best-response oracle, ii) we propose a modified projected replicated dynamics, i.e., R-PRD, which ensures the computed meta-strategy of the adversary fall in the threshold, and iii) we also propose a protocol for the few-shot learning of the multiple strategies during testing. Extensive experiments on MuJoCo environments demonstrate that our proposed methods can outperform existing baselines, e.g., MAML.

READ FULL TEXT
research
07/29/2022

Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis

Meta reinforcement learning (meta RL), as a combination of meta-learning...
research
02/19/2022

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

Robust Reinforcement Learning (RL) focuses on improving performances und...
research
05/25/2022

Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization

We tackle real-world problems with complex structures beyond the pixel-b...
research
08/09/2021

Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning

Recent studies in multi-agent communicative reinforcement learning (MACR...
research
08/04/2020

Robust Reinforcement Learning using Adversarial Populations

Reinforcement Learning (RL) is an effective tool for controller design b...
research
07/28/2023

Learning with Constraint Learning: New Perspective, Solution Strategy and Various Applications

The complexity of learning problems, such as Generative Adversarial Netw...
research
01/08/2021

Evolving Reinforcement Learning Algorithms

We propose a method for meta-learning reinforcement learning algorithms ...

Please sign up or login with your details

Forgot password? Click here to reset