DeepAI AI Chat
Log In Sign Up

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

by   Siliang Zeng, et al.

Multi-agent reinforcement learning (MARL) has attracted much research attention recently. However, unlike its single-agent counterpart, many theoretical and algorithmic aspects of MARL have not been well-understood. In this paper, we study the emergence of coordinated behavior by autonomous agents using an actor-critic (AC) algorithm. Specifically, we propose and analyze a class of coordinated actor-critic algorithms (CAC) in which individually parametrized policies have a shared part (which is jointly optimized among all agents) and a personalized part (which is only locally optimized). Such kind of partially personalized policy allows agents to learn to coordinate by leveraging peers' past experience and adapt to individual tasks. The flexibility in our design allows the proposed MARL-CAC algorithm to be used in a fully decentralized setting, where the agents can only communicate with their neighbors, as well as a federated setting, where the agents occasionally communicate with a server while optimizing their (partially personalized) local models. Theoretically, we show that under some standard regularity assumptions, the proposed MARL-CAC algorithm requires 𝒪(ϵ^-5/2) samples to achieve an ϵ-stationary solution (defined as the solution whose squared norm of the gradient of the objective function is less than ϵ). To the best of our knowledge, this work provides the first finite-sample guarantee for decentralized AC algorithm with partially personalized policies.


page 1

page 2

page 3

page 4


F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning

Traditional centralized multi-agent reinforcement learning (MARL) algori...

Decentralized Deterministic Multi-Agent Reinforcement Learning

[Zhang, ICML 2018] provided the first decentralized actor-critic algorit...

Multi-agent Natural Actor-critic Reinforcement Learning Algorithms

Both single-agent and multi-agent actor-critic algorithms are an importa...

Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

We consider the problem of fully decentralized multi-agent reinforcement...

Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes

Most hard attention models initially observe a complete scene to locate ...

Distributed Learning of Decentralized Control Policies for Articulated Mobile Robots

State-of-the-art distributed algorithms for reinforcement learning rely ...

Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

Actor-critic (AC) algorithms have been widely adopted in decentralized m...