Few-Shot Bayesian Optimization with Deep Kernel Surrogates

01/19/2021
by   Martin Wistuba, et al.
0

Hyperparameter optimization (HPO) is a central pillar in the automation of machine learning solutions and is mainly performed via Bayesian optimization, where a parametric surrogate is learned to approximate the black box response function (e.g. validation error). Unfortunately, evaluating the response function is computationally intensive. As a remedy, earlier work emphasizes the need for transfer learning surrogates which learn to optimize hyperparameters for an algorithm from other tasks. In contrast to previous work, we propose to rethink HPO as a few-shot learning problem in which we train a shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of a new task. We propose the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets. As a result, the novel few-shot optimization of our deep kernel surrogate leads to new state-of-the-art results at HPO compared to several recent methods on diverse metadata sets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/09/2023

Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets

Bayesian optimization (BO) is widely adopted in black-box optimization p...
research
07/18/2017

Robust Bayesian Optimization with Student-t Likelihood

Bayesian optimization has recently attracted the attention of the automa...
research
07/28/2016

Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates

Automatically searching for optimal hyperparameter configurations is of ...
research
05/23/2019

Bayesian Optimization over Sets

We propose a Bayesian optimization method over sets, to minimize a black...
research
09/06/2021

Towards Reusable Surrogate Models: Graph-Based Transfer Learning on Trusses

Surrogate models have several uses in engineering design, including spee...
research
10/31/2022

Optimizing Closed-Loop Performance with Data from Similar Systems: A Bayesian Meta-Learning Approach

Bayesian optimization (BO) has demonstrated potential for optimizing con...
research
06/08/2021

Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization

Bayesian optimization (BO) conventionally relies on handcrafted acquisit...

Please sign up or login with your details

Forgot password? Click here to reset