Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

12/03/2020
by   Michael Dennis, et al.
0

A wide range of reinforcement learning (RL) problems - including robustness, transfer learning, unsupervised RL, and emergent complexity - require specifying a distribution of tasks or environments in which a policy will be trained. However, creating a useful distribution of environments is error prone, and takes a significant amount of developer time and effort. We propose Unsupervised Environment Design (UED) as an alternative paradigm, where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. Existing approaches to automatically generating environments suffer from common failure modes: domain randomization cannot generate structure or adapt the difficulty of the environment to the agent's learning progress, and minimax adversarial training leads to worst-case environments that are often unsolvable. To generate structured, solvable environments for our protagonist agent, we introduce a second, antagonist agent that is allied with the environment-generating adversary. The adversary is motivated to generate environments which maximize regret, defined as the difference between the protagonist and antagonist agent's return. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.

READ FULL TEXT

page 2

page 8

page 9

page 18

research
03/02/2021

Adversarial Environment Generation for Learning to Navigate the Web

Learning to autonomously navigate the web is a difficult sequential deci...
research
10/06/2021

Replay-Guided Adversarial Environment Design

Deep reinforcement learning (RL) agents may successfully generalize to n...
research
01/19/2023

Effective Diversity in Unsupervised Environment Design

Agent decision making using Reinforcement Learning (RL) heavily relies o...
research
10/19/2022

CLUTR: Curriculum Learning via Unsupervised Task Representation Learning

Reinforcement Learning (RL) algorithms are often known for sample ineffi...
research
07/11/2022

Grounding Aleatoric Uncertainty in Unsupervised Environment Design

Adaptive curricula in reinforcement learning (RL) have proven effective ...
research
08/21/2023

Stabilizing Unsupervised Environment Design with a Learned Adversary

A key challenge in training generally-capable agents is the design of tr...
research
06/15/2023

Reward-Free Curricula for Training Robust World Models

There has been a recent surge of interest in developing generally-capabl...

Please sign up or login with your details

Forgot password? Click here to reset