Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing

03/05/2020
by   Hangyu Mao, et al.
0

In cooperative multi-agent reinforcement learning (MARL), how to design a suitable reward signal to accelerate learning and stabilize convergence is a critical problem. The global reward signal assigns the same global reward to all agents without distinguishing their contributions, while the local reward signal provides different local rewards to each agent based solely on individual behavior. Both of the two reward assignment approaches have some shortcomings: the former might encourage lazy agents, while the latter might produce selfish agents. In this paper, we study reward design problem in cooperative MARL based on packet routing environments. Firstly, we show that the above two reward signals are prone to produce suboptimal policies. Then, inspired by some observations and considerations, we design some mixed reward signals, which are off-the-shelf to learn better policies. Finally, we turn the mixed reward signals into the adaptive counterparts, which achieve best results in our experiments. Other reward signals are also discussed in this paper. As reward design is a very fundamental problem in RL and especially in MARL, we hope that MARL researchers can rethink the rewards used in their systems.

READ FULL TEXT
research
03/24/2020

Multi-Agent Reinforcement Learning for Problems with Combined Individual and Team Reward

Many cooperative multi-agent problems require agents to learn individual...
research
05/17/2021

Learning to Win, Lose and Cooperate through Reward Signal Evolution

Solving a reinforcement learning problem typically involves correctly pr...
research
08/24/2023

Predator-prey survival pressure is sufficient to evolve swarming behaviors

The comprehension of how local interactions arise in global collective b...
research
07/11/2019

Shapley Q-value: A Local Reward Approach to Solve Global Reward Games

Cooperative game is a critical research area in multi-agent reinforcemen...
research
03/24/2023

Learning Reward Machines in Cooperative Multi-Agent Tasks

This paper presents a novel approach to Multi-Agent Reinforcement Learni...
research
02/28/2023

On Learning Intrinsic Rewards for Faster Multi-Agent Reinforcement Learning based MAC Protocol Design in 6G Wireless Networks

In this paper, we propose a novel framework for designing a fast converg...
research
12/14/2020

Efficient Querying for Cooperative Probabilistic Commitments

Multiagent systems can use commitments as the core of a general coordina...

Please sign up or login with your details

Forgot password? Click here to reset