Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach

12/15/2021
by   Setareh Ariafar, et al.
0

Black box optimization requires specifying a search space to explore for solutions, e.g. a d-dimensional compact space, and this choice is critical for getting the best results at a reasonable budget. Unfortunately, determining a high quality search space can be challenging in many applications. For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable. The goal of this work is to motivate – through example applications in tuning deep neural networks – the problem of predicting the quality of search spaces conditioned on budgets, as well as to provide a simple scoring method based on a utility function applied to a probabilistic response surface model, similar to Bayesian optimization. We show that the method we present can compute meaningful budget-conditional scores in a variety of situations. We also provide experimental evidence that accurate scores can be useful in constructing and pruning search spaces. Ultimately, we believe scoring search spaces should become standard practice in the experimental workflow for deep learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/18/2019

A Simple Heuristic for Bayesian Optimization with A Low Budget

The aim of black-box optimization is to optimize an objective function w...
research
07/10/2023

SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees

Gradient boosted trees (GBTs) are ubiquitous models used by researchers,...
research
03/25/2022

LAMBDA: Covering the Solution Set of Black-Box Inequality by Search Space Quantization

Black-box functions are broadly used to model complex problems that prov...
research
07/01/2020

Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search

High dimensional black-box optimization has broad applications but remai...
research
05/25/2018

Parallel Architecture and Hyperparameter Search via Successive Halving and Classification

We present a simple and powerful algorithm for parallel black box optimi...
research
06/23/2018

An Improved Generic Bet-and-Run Strategy for Speeding Up Stochastic Local Search

A commonly used strategy for improving optimization algorithms is to res...
research
03/21/2017

Deep Learning for Explicitly Modeling Optimization Landscapes

In all but the most trivial optimization problems, the structure of the ...

Please sign up or login with your details

Forgot password? Click here to reset