Podracer architectures for scalable Reinforcement Learning

04/13/2021
by   Matteo Hessel, et al.
0

Supporting state-of-the-art AI research requires balancing rapid prototyping, ease of use, and quick iteration, with the ability to deploy experiments at a scale traditionally associated with production systems.Deep learning frameworks such as TensorFlow, PyTorch and JAX allow users to transparently make use of accelerators, such as TPUs and GPUs, to offload the more computationally intensive parts of training and inference in modern deep learning systems. Popular training pipelines that use these frameworks for deep learning typically focus on (un-)supervised learning. How to best train reinforcement learning (RL) agents at scale is still an active research area. In this report we argue that TPUs are particularly well suited for training RL agents in a scalable, efficient and reproducible way. Specifically we describe two architectures designed to make the best use of the resources available on a TPU Pod (a special configuration in a Google data center that features multiple TPU devices connected to each other by extremely low latency communication channels).

READ FULL TEXT

page 1

page 2

page 3

page 4

10/15/2019

SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference

We present a modern scalable reinforcement learning agent called SEED (S...
07/03/2021

Mava: a research framework for distributed multi-agent reinforcement learning

Breakthrough advances in reinforcement learning (RL) research have led t...
08/04/2020

EasyRL: A Simple and Extensible Reinforcement Learning Framework

In recent years, Reinforcement Learning (RL), has become a popular field...
06/11/2021

GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning

Deep Q Network (DQN) firstly kicked the door of deep reinforcement learn...
11/30/2019

IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks

The practical usage of reinforcement learning agents is often bottleneck...
02/04/2020

Neuro-evolutionary Frameworks for Generalized Learning Agents

The recent successes of deep learning and deep reinforcement learning ha...
03/02/2021

Sparse Training Theory for Scalable and Efficient Agents

A fundamental task for artificial intelligence is learning. Deep Neural ...