Virtual Replay Cache

12/06/2021
by   Brett Daley, et al.
0

Return caching is a recent strategy that enables efficient minibatch training with multistep estimators (e.g. the λ-return) for deep reinforcement learning. By precomputing return estimates in sequential batches and then storing the results in an auxiliary data structure for later sampling, the average computation spent per estimate can be greatly reduced. Still, the efficiency of return caching could be improved, particularly with regard to its large memory usage and repetitive data copies. We propose a new data structure, the Virtual Replay Cache (VRC), to address these shortcomings. When learning to play Atari 2600 games, the VRC nearly eliminates DQN(λ)'s cache memory footprint and slightly reduces the total training time on our hardware.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/15/2019

A Dual Memory Structure for Efficient Use of Replay Memory in Deep Reinforcement Learning

In this paper, we propose a dual memory structure for reinforcement lear...
research
10/03/2021

Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations

Reinforcement Learning (RL) has achieved significant success in applicat...
research
05/31/2023

AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-aware Prioritization

Multi-Agent Experience Replay (MER) is a key component of off-policy rei...
research
02/17/2022

CHEX: Multiversion Replay with Ordered Checkpoints

In scientific computing and data science disciplines, it is often necess...
research
10/18/2021

Data structure for node connectivity queries

Let κ(s,t) denote the maximum number of internally disjoint paths in an ...
research
08/04/2022

CheckSync: Using Runtime-Integrated Checkpoints to Achieve High Availability

CheckSync provides applications with high availability via runtime-integ...

Please sign up or login with your details

Forgot password? Click here to reset