What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

06/10/2020
by   Marcin Andrychowicz, et al.
0

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such “choices” in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

READ FULL TEXT

page 22

page 25

page 26

page 29

page 32

page 36

page 41

page 42

research
10/09/2020

Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning

Learning to locomote is one of the most common tasks in physics-based an...
research
06/01/2023

Improving and Benchmarking Offline Reinforcement Learning Algorithms

Recently, Offline Reinforcement Learning (RL) has achieved remarkable pr...
research
02/28/2019

Catalyst.RL: A Distributed Framework for Reproducible RL Research

Despite the recent progress in deep reinforcement learning field (RL), a...
research
06/02/2019

An Empirical Study on Hyperparameters and their Interdependence for RL Generalization

Recent results in Reinforcement Learning (RL) have shown that agents wit...
research
06/01/2020

Acme: A Research Framework for Distributed Reinforcement Learning

Deep reinforcement learning has led to many recent-and groundbreaking-ad...
research
11/03/2021

Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies

Reinforcement learning (RL) for continuous control typically employs dis...
research
09/24/2022

Unsupervised Model-based Pre-training for Data-efficient Control from Pixels

Controlling artificial agents from visual sensory data is an arduous tas...

Please sign up or login with your details

Forgot password? Click here to reset