Event Tables for Efficient Experience Replay

11/01/2022
by   Varun Kompella, et al.
0

Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), which partitions an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. We prove a theoretical advantage over the traditional monolithic buffer approach and combine SSET with an existing prioritized sampling strategy to further improve learning speed and stability. Empirical results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling approaches.

READ FULL TEXT

page 4

page 5

page 6

page 13

page 15

page 17

page 18

research
10/04/2021

Large Batch Experience Replay

Several algorithms have been proposed to sample non-uniformly the replay...
research
02/09/2021

Reverb: A Framework For Experience Replay

A central component of training in Reinforcement Learning (RL) is Experi...
research
06/16/2023

Temporal Difference Learning with Experience Replay

Temporal-difference (TD) learning is widely regarded as one of the most ...
research
10/01/2021

Sim and Real: Better Together

Simulation is used extensively in autonomous systems, particularly in ro...
research
09/24/2020

Dynamic Buffer Sizing for Out-of-Order Event Compensation for Time-Sensitive Applications

Today's sensor network implementations often comprise various types of n...
research
07/15/2023

An Empirical Study of the Effectiveness of Using a Replay Buffer on Mode Discovery in GFlowNets

Reinforcement Learning (RL) algorithms aim to learn an optimal policy by...
research
10/03/2021

Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations

Reinforcement Learning (RL) has achieved significant success in applicat...

Please sign up or login with your details

Forgot password? Click here to reset