DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization

05/20/2021
by   Noor Awad, et al.
15

Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever. Here, we combine the advantages of the popular bandit-based HPO method Hyperband (HB) and the evolutionary search approach of Differential Evolution (DE) to yield a new HPO method which we call DEHB. Comprehensive results on a very broad range of HPO problems, as well as a wide range of tabular benchmarks from neural architecture search, demonstrate that DEHB achieves strong performance far more robustly than all previous HPO methods we are aware of, especially for high-dimensional problems with discrete input dimensions. For example, DEHB is up to 1000x faster than random search. It is also efficient in computational time, conceptually simple and easy to implement, positioning it well to become a new default HPO method.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

07/04/2018

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Modern deep learning methods are very sensitive to many hyperparameters,...
04/03/2020

Weighted Random Search for Hyperparameter Optimization

We introduce an improved version of Random Search (RS), used here for hy...
11/23/2018

Learning Multiple Defaults for Machine Learning Algorithms

The performance of modern machine learning methods highly depends on the...
11/09/2020

Neural Architecture Search with an Efficient Multiobjective Evolutionary Framework

Deep learning methods have become very successful at solving many comple...
07/18/2018

Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

While existing work on neural architecture search (NAS) tunes hyperparam...
05/17/2013

Evolutionary optimization of an experimental apparatus

In recent decades, cold atom experiments have become increasingly comple...
08/14/2020

Efficient hyperparameter optimization by way of PAC-Bayes bound minimization

Identifying optimal values for a high-dimensional set of hyperparameters...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.