DeepOBS: A Deep Learning Optimizer Benchmark Suite

by   Frank Schneider, et al.

Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications. It supports TensorFlow and is available open source.


GuacaMol: Benchmarking Models for De Novo Molecular Design

De novo design seeks to generate molecules with required property profil...

CompModels: A suite of computer model test functions for Bayesian optimization

The CompModels package for R provides a suite of computer model test fun...

Descending through a Crowded Valley – Benchmarking Deep Learning Optimizers

Choosing the optimizer is among the most crucial decisions of deep learn...

Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking

Deep reinforcement learning has been one of the fastest growing fields o...

BENCHIP: Benchmarking Intelligence Processors

The increasing attention on deep learning has tremendously spurred the d...

On Empirical Comparisons of Optimizers for Deep Learning

Selecting an optimizer is a central step in the contemporary deep learni...

Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification

Data-efficient image classification using deep neural networks in settin...