DeepAI AI Chat
Log In Sign Up

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

10/11/2021
by   Siliang Zeng, et al.
13

Multi-agent reinforcement learning (MARL) has attracted much research attention recently. However, unlike its single-agent counterpart, many theoretical and algorithmic aspects of MARL have not been well-understood. In this paper, we study the emergence of coordinated behavior by autonomous agents using an actor-critic (AC) algorithm. Specifically, we propose and analyze a class of coordinated actor-critic algorithms (CAC) in which individually parametrized policies have a shared part (which is jointly optimized among all agents) and a personalized part (which is only locally optimized). Such kind of partially personalized policy allows agents to learn to coordinate by leveraging peers' past experience and adapt to individual tasks. The flexibility in our design allows the proposed MARL-CAC algorithm to be used in a fully decentralized setting, where the agents can only communicate with their neighbors, as well as a federated setting, where the agents occasionally communicate with a server while optimizing their (partially personalized) local models. Theoretically, we show that under some standard regularity assumptions, the proposed MARL-CAC algorithm requires 𝒪(ϵ^-5/2) samples to achieve an ϵ-stationary solution (defined as the solution whose squared norm of the gradient of the objective function is less than ϵ). To the best of our knowledge, this work provides the first finite-sample guarantee for decentralized AC algorithm with partially personalized policies.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/17/2020

F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning

Traditional centralized multi-agent reinforcement learning (MARL) algori...
02/19/2021

Decentralized Deterministic Multi-Agent Reinforcement Learning

[Zhang, ICML 2018] provided the first decentralized actor-critic algorit...
09/03/2021

Multi-agent Natural Actor-critic Reinforcement Learning Algorithms

Both single-agent and multi-agent actor-critic algorithms are an importa...
02/23/2018

Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

We consider the problem of fully decentralized multi-agent reinforcement...
04/01/2022

Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes

Most hard attention models initially observe a complete scene to locate ...
01/24/2019

Distributed Learning of Decentralized Control Policies for Articulated Mobile Robots

State-of-the-art distributed algorithms for reinforcement learning rely ...
09/08/2021

Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

Actor-critic (AC) algorithms have been widely adopted in decentralized m...