Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

11/08/2021
by   Jiayu Chen, et al.
4

We introduce a curriculum learning algorithm, Variational Automatic Curriculum Learning (VACL), for solving challenging goal-conditioned cooperative multi-agent reinforcement learning problems. We motivate our paradigm through a variational perspective, where the learning objective can be decomposed into two terms: task learning on the current task distribution, and curriculum update to a new task distribution. Local optimization over the second term suggests that the curriculum should gradually expand the training tasks from easy to hard. Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity progression, which produces training curricula over both the task configurations as well as the number of entities in the task. Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents. Particularly, using a single desktop machine, VACL achieves 98 with 100 agents in the simple-spread benchmark and reproduces the ramp-use behavior originally shown in OpenAI's hide-and-seek project. Our project website is at https://sites.google.com/view/vacl-neurips-2021.

READ FULL TEXT

page 5

page 9

page 10

research
02/07/2023

Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

Recent advances in multi-agent reinforcement learning (MARL) allow agent...
research
08/02/2020

Curriculum Learning with a Progression Function

Curriculum Learning for Reinforcement Learning is an increasingly popula...
research
05/20/2022

Self-Paced Multi-Agent Reinforcement Learning

Curriculum reinforcement learning (CRL) aims to speed up learning of a t...
research
03/23/2020

Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

In multi-agent games, the complexity of the environment can grow exponen...
research
12/06/2022

Curriculum Learning for Relative Overgeneralization

In multi-agent reinforcement learning (MARL), many popular methods, such...
research
03/10/2020

Automatic Curriculum Learning For Deep RL: A Short Survey

Automatic Curriculum Learning (ACL) has become a cornerstone of recent s...

Please sign up or login with your details

Forgot password? Click here to reset