Multi-Agent MDP Homomorphic Networks

10/09/2021
by   Elise van der Pol, et al.
0

This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state-action space of cooperative multi-agent systems. In cooperative multi-agent systems, complex symmetries arise between different configurations of the agents and their local observations. For example, consider a group of agents navigating: rotating the state globally results in a permutation of the optimal joint policy. Existing work on symmetries in single agent reinforcement learning can only be generalized to the fully centralized setting, because such approaches rely on the global symmetry in the full state-action spaces, and these can result in correspondences across agents. To encode such symmetries while still allowing distributed execution we propose a factorization that decomposes global symmetries into local transformations. Our proposed factorization allows for distributing the computation that enforces global symmetries over local agents and local interactions. We introduce a multi-agent equivariant policy network based on this factorization. We show empirically on symmetric multi-agent problems that distributed execution of globally symmetric policies improves data efficiency compared to non-equivariant baselines.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/12/2022

Towards Global Optimality in Cooperative MARL with Sequential Transformation

Policy learning in multi-agent reinforcement learning (MARL) is challeng...
research
12/22/2020

Distributed Q-Learning with State Tracking for Multi-agent Networked Control

This paper studies distributed Q-learning for Linear Quadratic Regulator...
research
06/25/2013

Distributed Heuristic Forward Search for Multi-Agent Systems

This paper describes a number of distributed forward search algorithms f...
research
11/04/2014

Learning of Agent Capability Models with Applications in Multi-agent Planning

One important challenge for a set of agents to achieve more efficient co...
research
11/19/2020

Zeroth-Order Feedback Optimization for Cooperative Multi-Agent Systems

We study a class of cooperative multi-agent optimization problems, where...
research
02/08/2023

Efficient Planning in Combinatorial Action Spaces with Applications to Cooperative Multi-Agent Reinforcement Learning

A practical challenge in reinforcement learning are combinatorial action...
research
10/22/2021

Convergence Rates of Average-Reward Multi-agent Reinforcement Learning via Randomized Linear Programming

In tabular multi-agent reinforcement learning with average-cost criterio...

Please sign up or login with your details

Forgot password? Click here to reset