Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures

09/19/2021
by   Chapman Siu, et al.
10

We propose using regularization for Multi-Agent Reinforcement Learning rather than learning explicit cooperative structures called Multi-Agent Regularized Q-learning (MARQ). Many MARL approaches leverage centralized structures in order to exploit global state information or removing communication constraints when the agents act in a decentralized manner. Instead of learning redundant structures which is removed during agent execution, we propose instead to leverage shared experiences of the agents to regularize the individual policies in order to promote structured exploration. We examine several different approaches to how MARQ can either explicitly or implicitly regularize our policies in a multi-agent setting. MARQ aims to address these limitations in the MARL context through applying regularization constraints which can correct bias in off-policy out-of-distribution agent experiences and promote diverse exploration. Our algorithm is evaluated on several benchmark multi-agent environments and we show that MARQ consistently outperforms several baselines and state-of-the-art algorithms; learning in fewer steps and converging to higher returns.

READ FULL TEXT

page 6

page 8

page 10

page 11

research
05/27/2023

Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?

Centralized Training with Decentralized Execution (CTDE) has recently em...
research
10/17/2022

PTDE: Personalized Training with Distillated Execution for Multi-Agent Reinforcement Learning

Centralized Training with Decentralized Execution (CTDE) has been a very...
research
01/22/2020

On Solving Cooperative MARL Problems with a Few Good Experiences

Cooperative Multi-agent Reinforcement Learning (MARL) is crucial for coo...
research
09/02/2023

Multi-agent Collective Construction using 3D Decomposition

This paper addresses a Multi-Agent Collective Construction (MACC) proble...
research
05/25/2022

Scalable Multi-Agent Model-Based Reinforcement Learning

Recent Multi-Agent Reinforcement Learning (MARL) literature has been lar...
research
08/21/2020

Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication

With the rise of online e-commerce platforms, more and more customers pr...
research
10/16/2019

MAVEN: Multi-Agent Variational Exploration

Centralised training with decentralised execution is an important settin...

Please sign up or login with your details

Forgot password? Click here to reset