How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

06/21/2018

∙

by   Cédric Colas, et al.

∙

4

∙

share

Consistently checking the statistical significance of experimental results is one of the mandatory methodological steps to address the so-called "reproducibility crisis" in deep reinforcement learning. In this tutorial paper, we explain how to determine the number of random seeds one should use to provide a statistically significant comparison of the performance of two algorithms. We also discuss the influence of deviations from the assumptions usually made by statistical tests, we provide guidelines to counter their negative effects and some code to perform the tests.