Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning

07/29/2020
by   Lars Hertel, et al.
16

Reinforcement learning algorithms can show strong variation in performance between training runs with different random seeds. In this paper we explore how this affects hyperparameter optimization when the goal is to find hyperparameter settings that perform well across random seeds. In particular, we benchmark whether it is better to explore a large quantity of hyperparameter settings via pruning of bad performers, or if it is better to aim for quality of collected results by using repetitions. For this we consider the Successive Halving, Random Search, and Bayesian Optimization algorithms, the latter two with and without repetitions. We apply these to tuning the PPO2 algorithm on the Cartpole balancing task and the Inverted Pendulum Swing-up task. We demonstrate that pruning may negatively affect the optimization and that repeated sampling does not help in finding hyperparameter settings that perform better across random seeds. From our experiments we conclude that Bayesian optimization with a noise robust acquisition function is the best choice for hyperparameter optimization in reinforcement learning tasks.

READ FULL TEXT
research
03/21/2020

Towards Automatic Bayesian Optimization: A first step involving acquisition functions

Bayesian Optimization is the state of the art technique for the optimiza...
research
10/05/2020

Non-trivial informational closure of a Bayesian hyperparameter

We investigate the non-trivial informational closure (NTIC) of a Bayesia...
research
04/26/2021

Efficient Hyperparameter Optimization for Physics-based Character Animation

Physics-based character animation has seen significant advances in recen...
research
09/20/2019

Bayesian Optimization for Iterative Learning

The success of deep (reinforcement) learning systems crucially depends o...
research
03/10/2020

Explore and Exploit with Heterotic Line Bundle Models

We use deep reinforcement learning to explore a class of heterotic SU(5)...
research
12/23/2021

Using Sequential Statistical Tests to Improve the Performance of Random Search in hyperparameter Tuning

Hyperparamter tuning is one of the the most time-consuming parts in mach...

Please sign up or login with your details

Forgot password? Click here to reset