DeepAI AI Chat
Log In Sign Up

MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

by   Kefan Su, et al.

Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in decentralized learning. In the paper, we tackle the non-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns to update their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees a ε-convergence at each turn, their joint policy converges to a Nash equilibrium. In practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL on a variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.


page 1

page 2

page 3

page 4


Best Possible Q-Learning

Fully decentralized learning, where the global information, i.e., the ac...

Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration

Many reality tasks such as robot coordination can be naturally modelled ...

Deep Decentralized Reinforcement Learning for Cooperative Control

In order to collaborate efficiently with unknown partners in cooperative...

Attention-Based Recurrence for Multi-Agent Reinforcement Learning under State Uncertainty

State uncertainty poses a major challenge for decentralized coordination...

Decentralized Reinforcement Learning: Global Decision-Making via Local Economic Transactions

This paper seeks to establish a framework for directing a society of sim...

BayGo: Joint Bayesian Learning and Information-Aware Graph Optimization

This article deals with the problem of distributed machine learning, in ...

Rethinking Individual Global Max in Cooperative Multi-Agent Reinforcement Learning

In cooperative multi-agent reinforcement learning, centralized training ...