One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL

10/27/2020
by   Saurabh Kumar, et al.
4

While reinforcement learning algorithms can learn effective policies for complex tasks, these policies are often brittle to even minor task variations, especially when variations are not explicitly provided during training. One natural approach to this problem is to train agents with manually specified variation in the training task or environment. However, this may be infeasible in practical situations, either because making perturbations is not possible, or because it is unclear how to choose suitable perturbation strategies without sacrificing performance. The key insight of this work is that learning diverse behaviors for accomplishing a task can directly lead to behavior that generalizes to varying environments, without needing to perform explicit perturbations during training. By identifying multiple solutions for the task in a single environment during training, our approach can generalize to new situations by abandoning solutions that are no longer effective and adopting those that are. We theoretically characterize a robustness set of environments that arises from our algorithm and empirically find that our diversity-driven approach can extrapolate to various changes in the environment and task.

READ FULL TEXT
research
09/24/2022

Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation

While Reinforcement Learning can achieve impressive results for complex ...
research
10/11/2021

Learning a subspace of policies for online adaptation in Reinforcement Learning

Deep Reinforcement Learning (RL) is mainly studied in a setting where th...
research
07/02/2013

Multi-Task Policy Search

Learning policies that generalize across multiple tasks is an important ...
research
02/14/2022

Robust Policy Learning over Multiple Uncertainty Sets

Reinforcement learning (RL) agents need to be robust to variations in sa...
research
03/12/2021

Discovering Diverse Solutions in Deep Reinforcement Learning

Reinforcement learning (RL) algorithms are typically limited to learning...
research
04/05/2019

Synthesized Policies for Transfer and Adaptation across Tasks and Environments

The ability to transfer in reinforcement learning is key towards buildin...
research
03/15/2022

Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning

Assembly of multi-part physical structures is both a valuable end produc...

Please sign up or login with your details

Forgot password? Click here to reset