Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

10/19/2018
by   Natasha Jaques, et al.
0

We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions that it could have taken, and computes their effect on the behavior of other agents. Actions that lead to bigger changes in other agents' behavior are considered influential and are rewarded. We show that this is equivalent to rewarding agents for having high mutual information between their actions. Empirical results demonstrate that influence leads to enhanced coordination and communication in challenging social dilemma environments, dramatically increasing the learning curves of the deep RL agents, and leading to more meaningful learned communication protocols. The influence rewards for all agents can be computed in a decentralized way by enabling agents to learn a model of other agents using deep neural networks. In contrast, key previous works on emergent communication in the MARL setting were unable to learn diverse policies in a decentralized manner and had to resort to centralized training. Consequently, the influence reward opens up a window of new opportunities for research in this area.

READ FULL TEXT
10/19/2018

Intrinsic Social Motivation via Causal Influence in Multi-Agent RL

We derive a new intrinsic social motivation for multi-agent reinforcemen...
12/15/2020

Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication

We consider the problem of robust multi-agent reinforcement learning (MA...
03/07/2022

Reliably Re-Acting to Partner's Actions with the Social Intrinsic Motivation of Transfer Empowerment

We consider multi-agent reinforcement learning (MARL) for cooperative co...
09/26/2018

Learning through Probing: a decentralized reinforcement learning architecture for social dilemmas

Multi-agent reinforcement learning has received significant interest in ...
04/21/2022

Path-Specific Objectives for Safer Agent Incentives

We present a general framework for training safe agents whose naive ince...
02/16/2018

Learning multiagent coordination in the absence of communication channels

In this work, we develop a reinforcement learning protocol for a multiag...
06/21/2021

Curriculum-Driven Multi-Agent Learning and the Role of Implicit Communication in Teamwork

We propose a curriculum-driven learning strategy for solving difficult m...

Code Repositories

sequential_social_dilemma_games

Repo for reproduction of sequential social dilemmas


view repo