Selective Experience Replay for Lifelong Learning

02/28/2018
by   David Isele, et al.
0

Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an experience replay process that augments the standard FIFO buffer and selectively stores experiences in a long-term memory. We explore four strategies for selecting which experiences will be stored: favoring surprise, favoring reward, matching the global training distribution, and maximizing coverage of the state space. We show that distribution matching successfully prevents catastrophic forgetting, and is consistently the best approach on all domains tested. While distribution matching has better and more consistent performance, we identify one case in which coverage maximization is beneficial - when tasks that receive less trained are more important. Overall, our results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting.

READ FULL TEXT

page 5

page 6

page 7

research
05/22/2022

Memory-efficient Reinforcement Learning with Knowledge Consolidation

Artificial neural networks are promising as general function approximato...
research
08/09/2022

Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2

One approach to meet the challenges of deep lifelong reinforcement learn...
research
09/18/2023

Stabilizing RLHF through Advantage Model and Selective Rehearsal

Large Language Models (LLMs) have revolutionized natural language proces...
research
05/03/2023

Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

Deep Reinforcement Learning agents often suffer from catastrophic forget...
research
02/22/2023

Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging

Selective experience replay is a popular strategy for integrating lifelo...
research
10/03/2022

How Relevant is Selective Memory Population in Lifelong Language Learning?

Lifelong language learning seeks to have models continuously learn multi...
research
11/17/2017

Generation and Consolidation of Recollections for Efficient Deep Lifelong Learning

Deep lifelong learning systems need to efficiently manage resources to s...

Please sign up or login with your details

Forgot password? Click here to reset