UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios

03/28/2022
by   Jiajun Chai, et al.
0

Multi-agent reinforcement learning methods such as VDN, QMIX, and QTRAN that adopt centralized training with decentralized execution (CTDE) framework have shown promising results in cooperation and competition. However, in some multi-agent scenarios, the number of agents and the size of action set actually vary over time. We call these unshaped scenarios, and the methods mentioned above fail in performing satisfyingly. In this paper, we propose a new method called Unshaped Networks for Multi-Agent Systems (UNMAS) that adapts to the number and size changes in multi-agent systems. We propose the self-weighting mixing network to factorize the joint action-value. Its adaption to the change in agent number is attributed to the nonlinear mapping from each-agent Q value to the joint action-value with individual weights. Besides, in order to address the change in action set, each agent constructs an individual action-value network that is composed of two streams to evaluate the constant environment-oriented subset and the varying unit-oriented subset. We evaluate UNMAS on various StarCraft II micro-management scenarios and compare the results with several state-of-the-art MARL algorithms. The superiority of UNMAS is demonstrated by its highest winning rates especially on the most difficult scenario 3s5z_vs_3s6z. The agents learn to perform effectively cooperative behaviors while other MARL algorithms fail in. Animated demonstrations and source code are provided in https://sites.google.com/view/unmas.

READ FULL TEXT

page 1

page 6

page 9

page 10

page 12

research
08/15/2022

Transformer-based Value Function Decomposition for Cooperative Multi-agent Reinforcement Learning in StarCraft

The StarCraft II Multi-Agent Challenge (SMAC) was created to be a challe...
research
05/30/2022

Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) is useful in many problems tha...
research
12/09/2021

Value Function Factorisation with Hypergraph Convolution for Cooperative Multi-agent Reinforcement Learning

Cooperation between agents in a multi-agent system (MAS) has become a ho...
research
09/19/2021

Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning

This paper introduces Greedy UnMix (GUM) for cooperative multi-agent rei...
research
10/26/2021

Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization

Self-Driven Particles (SDP) describe a category of multi-agent systems c...
research
06/15/2022

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

Many advances in cooperative multi-agent reinforcement learning (MARL) a...
research
12/14/2022

Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning

Adequate strategizing of agents behaviors is essential to solving cooper...

Please sign up or login with your details

Forgot password? Click here to reset