Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

04/25/2019
by   Tom Schaul, et al.
24

Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms. We study the learning dynamics of reinforcement learning (RL), specifically a characteristic coupling between learning and data generation that arises because RL agents control their future data distribution. In the presence of function approximation, this coupling can lead to a problematic type of 'ray interference', characterized by learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn one thing at a time even when learning in parallel is better. We establish the conditions under which ray interference occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting. We characterize a number of its properties and discuss possible remedies.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/26/2017

Ray RLLib: A Composable and Scalable Reinforcement Learning Library

Reinforcement learning (RL) algorithms involve the deep nesting of disti...
research
08/04/2023

Deep Reinforcement Learning for Autonomous Spacecraft Inspection using Illumination

This paper investigates the problem of on-orbit spacecraft inspection us...
research
09/04/2019

Learning sparse representations in reinforcement learning

Reinforcement learning (RL) algorithms allow artificial agents to improv...
research
02/28/2020

On Catastrophic Interference in Atari 2600 Games

Model-free deep reinforcement learning algorithms are troubled with poor...
research
06/11/2021

GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning

Deep Q Network (DQN) firstly kicked the door of deep reinforcement learn...
research
05/11/2021

Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective

Most of the recent deep reinforcement learning advances take an RL-centr...
research
12/14/2021

Representation and Invariance in Reinforcement Learning

If we changed the rules, would the wise trade places with the fools? Dif...

Please sign up or login with your details

Forgot password? Click here to reset