Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning

05/31/2020
by   Jianhao Wang, et al.
0

Value decomposition is a popular and promising approach to scaling up multi-agent reinforcement learning in cooperative settings. However, the theoretical understanding of such methods is limited. In this paper, we introduce a variant of the fitted Q-iteration framework for analyzing multi-agent Q-learning with value decomposition. Based on this framework, we derive a closed-form solution to the Bellman error minimization with linear value decomposition. With this novel solution, we further reveal two interesting insights: 1) linear value decomposition implicitly implements a classical multi-agent credit assignment called counterfactual difference rewards; and 2) multi-agent Q-learning with linear value decomposition requires on-policy data distribution to achieve numerical stability. In the empirical study, our experiments demonstrate the realizability of our theoretical implications in a broad set of complicated tasks. They show that most state-of-the-art deep multi-agent Q-learning algorithms using linear value decomposition cannot efficiently utilize off-policy samples, which may even lead to an unbounded divergence.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/04/2023

Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learning

Value decomposition methods have gradually become popular in the coopera...
research
12/07/2020

Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation

Cooperative multi-agent tasks require agents to deduce their own contrib...
research
08/07/2022

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

We explore value decomposition solutions for multi-agent deep reinforcem...
research
01/25/2022

Multi-agent Performative Prediction: From Global Stability and Optimality to Chaos

The recent framework of performative prediction is aimed at capturing se...
research
05/13/2021

SIDE: I Infer the State I Want to Learn

As one of the solutions to the Dec-POMDP problem, the value decompositio...
research
02/10/2020

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

In many real-world settings, a team of cooperative agents must learn to ...
research
06/15/2022

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

Many advances in cooperative multi-agent reinforcement learning (MARL) a...

Please sign up or login with your details

Forgot password? Click here to reset