Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark

by   Sharada Mohanty, et al.

The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way. We designed the benchmark on top of the already existing Procgen Benchmark by defining clear tasks and standardizing the end to end evaluation setups. The design aims to maximize the flexibility available for researchers who wish to design future iterations of such benchmarks, and yet imposes necessary practical constraints to allow for a system like this to scale. This paper presents the competition setup and the details and analysis of the top solutions identified through this setup in context of 2020 iteration of the competition at NeurIPS.


page 4

page 5

page 7

page 8

page 23

page 24

page 25

page 26


Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning

To facilitate research in the direction of sample-efficient reinforcemen...

Leveraging Procedural Generation to Benchmark Reinforcement Learning

In this report, we introduce Procgen Benchmark, a suite of 16 procedural...

Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning

The idea of transfer in reinforcement learning (TRL) is intriguing: bein...

Towards robust and domain agnostic reinforcement learning competitions

Reinforcement learning competitions have formed the basis for standard r...

DeepTraffic: Driving Fast through Dense Traffic with Deep Reinforcement Learning

We present a micro-traffic simulation (named "DeepTraffic") where the pe...

SARS-CoV-2 Coronavirus Data Compression Benchmark

This paper introduces a lossless data compression competition that bench...

BitTensor: An Intermodel Intelligence Measure

A purely inter-model version of a machine intelligence benchmark would a...

Please sign up or login with your details

Forgot password? Click here to reset