Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

02/07/2023
by   Rundong Wang, et al.
0

Recent advances in multi-agent reinforcement learning (MARL) allow agents to coordinate their behaviors in complex environments. However, common MARL algorithms still suffer from scalability and sparse reward issues. One promising approach to resolving them is automatic curriculum learning (ACL). ACL involves a student (curriculum learner) training on tasks of increasing difficulty controlled by a teacher (curriculum generator). Despite its success, ACL's applicability is limited by (1) the lack of a general student framework for dealing with the varying number of agents across tasks and the sparse reward problem, and (2) the non-stationarity of the teacher's task due to ever-changing student strategies. As a remedy for ACL, we introduce a novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), which adapts curriculum learning to multi-agent coordination. Specifically, we endow the student with population-invariant communication and a hierarchical skill set, allowing it to learn cooperation and behavior skills from distinct tasks with varying numbers of agents. In addition, we model the teacher as a contextual bandit conditioned by student policies, enabling a team of agents to change its size while still retaining previously acquired skills. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem and provide a corresponding regret bound. Empirical results show that our method improves the performance, scalability and sample efficiency in several MARL environments.

READ FULL TEXT

page 6

page 8

research
07/01/2017

Teacher-Student Curriculum Learning

We propose Teacher-Student Curriculum Learning (TSCL), a framework for a...
research
11/08/2021

Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

We introduce a curriculum learning algorithm, Variational Automatic Curr...
research
08/24/2023

CGMI: Configurable General Multi-Agent Interaction Framework

Benefiting from the powerful capabilities of large language models (LLMs...
research
05/20/2022

Self-Paced Multi-Agent Reinforcement Learning

Curriculum reinforcement learning (CRL) aims to speed up learning of a t...
research
10/18/2017

Visual Progression Analysis of Student Records Data

University curriculum, both on a campus level and on a per-major level, ...
research
03/23/2020

Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

In multi-agent games, the complexity of the environment can grow exponen...
research
08/14/2020

Mastering Rate based Curriculum Learning

Recent automatic curriculum learning algorithms, and in particular Teach...

Please sign up or login with your details

Forgot password? Click here to reset