Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning

by   Sergio Valcarcel Macua, et al.

We propose a fully distributed actor-critic architecture, named Diff-DAC, with application to multitask reinforcement learning (MRL). During the learning process, agents communicate their value and policy parameters to their neighbours, diffusing the information across a network of agents with no need for a central station. Each agent can only access data from its local task, but aims to learn a common policy that performs well for the whole set of tasks. The architecture is scalable, since the computational and communication cost per agent depends on the number of neighbours rather than the overall number of agents. We derive Diff-DAC from duality theory and provide novel insights into the actor-critic framework, showing that it is actually an instance of the dual ascent method. We prove almost sure convergence of Diff-DAC to a common policy under general assumptions that hold even for deep-neural network approximations. For more restrictive assumptions, we also prove that this common policy is a stationary point of an approximation of the original problem. Numerical results on multitask extensions of common continuous control benchmarks demonstrate that Diff-DAC stabilises learning and has a regularising effect that induces higher performance and better generalisation properties than previous architectures.



There are no comments yet.


page 1

page 2

page 3

page 4


Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning

We propose a fully distributed actor-critic algorithm approximated by de...

Modular Multitask Reinforcement Learning with Policy Sketches

We describe a framework for multitask deep reinforcement learning guided...

Using Distributed Reinforcement Learning for Resource Orchestration in a Network Slicing Scenario

The Network Slicing (NS) paradigm enables the partition of physical and ...

Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control

Current Deep Reinforcement Learning algorithms still heavily rely on han...

Seq2Seq Mimic Games: A Signaling Perspective

We study the emergence of communication in multiagent adversarial settin...

Lyapunov-Based Reinforcement Learning State Estimator

In this paper, we consider the state estimation problem for nonlinear st...

Robust Dual View Deep Agent

Motivated by recent advance of machine learning using Deep Reinforcement...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.