A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning

01/03/2022
by   Xueguang Lyu, et al.
0

Centralized Training for Decentralized Execution, where training is done in a centralized offline fashion, has become a popular solution paradigm in Multi-Agent Reinforcement Learning. Many such methods take the form of actor-critic with state-based critics, since centralized training allows access to the true system state, which can be useful during training despite not being available at execution time. State-based critics have become a common empirical choice, albeit one which has had limited theoretical justification or analysis. In this paper, we show that state-based critics can introduce bias in the policy gradient estimates, potentially undermining the asymptotic guarantees of the algorithm. We also show that, even if the state-based critics do not introduce any bias, they can still result in a larger gradient variance, contrary to the common intuition. Finally, we show the effects of the theories in practice by comparing different forms of centralized critics on a wide range of common benchmarks, and detail how various environmental properties are related to the effectiveness of different types of critics.

READ FULL TEXT
research
02/08/2021

Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning

Centralized Training for Decentralized Execution, where agents are train...
research
10/17/2022

PTDE: Personalized Training with Distillated Execution for Multi-Agent Reinforcement Learning

Centralized Training with Decentralized Execution (CTDE) has been a very...
research
09/20/2022

Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning

Synchronizing decisions across multiple agents in realistic settings is ...
research
08/24/2023

An Efficient Distributed Multi-Agent Reinforcement Learning for EV Charging Network Control

The increasing trend in adopting electric vehicles (EVs) will significan...
research
06/15/2022

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

Many advances in cooperative multi-agent reinforcement learning (MARL) a...
research
01/04/2023

Attention-Based Recurrence for Multi-Agent Reinforcement Learning under State Uncertainty

State uncertainty poses a major challenge for decentralized coordination...
research
05/09/2023

Latent Interactive A2C for Improved RL in Open Many-Agent Systems

There is a prevalence of multiagent reinforcement learning (MARL) method...

Please sign up or login with your details

Forgot password? Click here to reset