Log In Sign Up

Difference Rewards Policy Gradients

by   Jacopo Castellini, et al.

Policy gradient methods have become one of the most popular classes of algorithms for multi-agent reinforcement learning. A key challenge, however, that is not addressed by many of these methods is multi-agent credit assignment: assessing an agent's contribution to the overall performance, which is crucial for learning good policies. We propose a novel algorithm called Dr.Reinforce that explicitly tackles this by combining difference rewards with policy gradients to allow for learning decentralized policies when the reward function is known. By differencing the reward function directly, Dr.Reinforce avoids difficulties associated with learning the Q-function as done by Counterfactual Multiagent Policy Gradients (COMA), a state-of-the-art difference rewards method. For applications where the reward function is unknown, we show the effectiveness of a version of Dr.Reinforce that learns an additional reward network that is used to estimate the difference rewards.


Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning

This paper proposes a definition of system health in the context of mult...

Learning Novel Policies For Tasks

In this work, we present a reinforcement learning algorithm that can fin...

Solving the Diffusion of Responsibility Problem in Multiagent Reinforcement Learning with a Policy Resonance Approach

SOTA multiagent reinforcement algorithms distinguish themselves in many ...

Leveraging Reward Gradients For Reinforcement Learning in Differentiable Physics Simulations

In recent years, fully differentiable rigid body physics simulators have...

Learning Human Objectives by Evaluating Hypothetical Behavior

We seek to align agent behavior with a user's objectives in a reinforcem...

Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning

Reward decomposition is a critical problem in centralized training with ...

Reward Engineering for Object Pick and Place Training

Robotic grasping is a crucial area of research as it can result in the a...