Scalable Multi-Agent Model-Based Reinforcement Learning

05/25/2022
by   Vladimir Egorov, et al.
0

Recent Multi-Agent Reinforcement Learning (MARL) literature has been largely focused on Centralized Training with Decentralized Execution (CTDE) paradigm. CTDE has been a dominant approach for both cooperative and mixed environments due to its capability to efficiently train decentralized policies. While in mixed environments full autonomy of the agents can be a desirable outcome, cooperative environments allow agents to share information to facilitate coordination. Approaches that leverage this technique are usually referred as communication methods, as full autonomy of agents is compromised for better performance. Although communication approaches have shown impressive results, they do not fully leverage this additional information during training phase. In this paper, we propose a new method called MAMBA which utilizes Model-Based Reinforcement Learning (MBRL) to further leverage centralized training in cooperative environments. We argue that communication between agents is enough to sustain a world model for each agent during execution phase while imaginary rollouts can be used for training, removing the necessity to interact with the environment. These properties yield sample efficient algorithm that can scale gracefully with the number of agents. We empirically confirm that MAMBA achieves good performance while reducing the number of interactions with the environment up to an orders of magnitude compared to Model-Free state-of-the-art approaches in challenging domains of SMAC and Flatland.

READ FULL TEXT

page 6

page 7

research
06/06/2022

Consensus Learning for Cooperative Multi-Agent Reinforcement Learning

Almost all multi-agent reinforcement learning algorithms without communi...
research
10/17/2022

PTDE: Personalized Training with Distillated Execution for Multi-Agent Reinforcement Learning

Centralized Training with Decentralized Execution (CTDE) has been a very...
research
09/19/2021

Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures

We propose using regularization for Multi-Agent Reinforcement Learning r...
research
11/28/2021

Evaluating Generalization and Transfer Capacity of Multi-Agent Reinforcement Learning Across Variable Number of Agents

Multi-agent Reinforcement Learning (MARL) problems often require coopera...
research
02/10/2020

Q-value Path Decomposition for Deep Multiagent Reinforcement Learning

Recently, deep multiagent reinforcement learning (MARL) has become a hig...
research
06/20/2023

IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL

We introduce IMP-MARL, an open-source suite of multi-agent reinforcement...
research
08/04/2022

Transferable Multi-Agent Reinforcement Learning with Dynamic Participating Agents

We study multi-agent reinforcement learning (MARL) with centralized trai...

Please sign up or login with your details

Forgot password? Click here to reset