A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning

03/01/2023
by   Woojun Kim, et al.
0

In this paper, we propose a new mutual information framework for multi-agent reinforcement learning to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the simultaneous mutual information between multi-agent actions. By introducing a latent variable to induce nonzero mutual information between multi-agent actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. The derived tractable objective can be interpreted as maximum entropy reinforcement learning combined with uncertainty reduction of other agents actions. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic, which follows centralized learning with decentralized execution. We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARL algorithms in multi-agent tasks requiring high-quality coordination.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/04/2020

A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning

In this paper, we propose a maximum mutual information (MMI) framework f...
research
05/17/2019

A Regularized Opponent Model with Maximum Entropy Objective

In a single-agent setting, reinforcement learning (RL) tasks can be cast...
research
11/12/2019

Learning Representations in Reinforcement Learning:An Information Bottleneck Approach

The information bottleneck principle is an elegant and useful approach t...
research
09/29/2015

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

The mutual information is a core statistical quantity that has applicati...
research
05/23/2022

Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning

Learning to coordinate is a daunting problem in multi-agent reinforcemen...
research
03/16/2022

PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration

Learning to collaborate is critical in Multi-Agent Reinforcement Learnin...
research
08/31/2021

APS: Active Pretraining with Successor Features

We introduce a new unsupervised pretraining objective for reinforcement ...

Please sign up or login with your details

Forgot password? Click here to reset