Using a thousand optimization tasks to learn hyperparameter search strategies

02/27/2020
by   Luke Metz, et al.
71

We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets. As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as  29 million training curves for these problems and the corresponding hyperparameters.

READ FULL TEXT

page 24

page 25

page 26

page 27

page 29

page 30

research
06/06/2017

Random Search for Hyperparameters using Determinantal Point Processes

We propose the use of k-determinantal point processes in hyperparameter ...
research
03/30/2020

Weighted Random Search for CNN Hyperparameter Optimization

Nearly all model algorithms used in machine learning use two different s...
research
06/02/2021

A Generalizable Approach to Learning Optimizers

A core issue with learning to optimize neural networks has been the lack...
research
06/06/2023

Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How

With the ever-increasing number of pretrained models, machine learning p...
research
10/12/2017

Hyperparameter Importance Across Datasets

With the advent of automated machine learning, automated hyperparameter ...
research
07/20/2020

Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization

Hyperparameters play a critical role in the performances of many machine...
research
07/06/2022

Model Agnostic Conformal Hyperparameter Optimization

Several novel frameworks for hyperparameter search have emerged in the l...

Please sign up or login with your details

Forgot password? Click here to reset