Log In Sign Up

Asynchronous Distributed Bayesian Optimization at HPC Scale

by   Romain Egele, et al.

Bayesian optimization (BO) is a widely used approach for computationally expensive black-box optimization such as simulator calibration and hyperparameter optimization of deep learning methods. In BO, a dynamically updated computationally cheap surrogate model is employed to learn the input-output relationship of the black-box function; this surrogate model is used to explore and exploit the promising regions of the input space. Multipoint BO methods adopt a single manager/multiple workers strategy to achieve high-quality solutions in shorter time. However, the computational overhead in multipoint generation schemes is a major bottleneck in designing BO methods that can scale to thousands of workers. We present an asynchronous-distributed BO (ADBO) method wherein each worker runs a search and asynchronously communicates the input-output values of black-box evaluations from all other workers without the manager. We scale our method up to 4,096 workers and demonstrate improvement in the quality of the solution and faster convergence. We demonstrate the effectiveness of our approach for tuning the hyperparameters of neural networks from the Exascale computing project CANDLE benchmarks.


page 1

page 2

page 3

page 4


Asynchronous ε-Greedy Bayesian Optimisation

Bayesian Optimisation (BO) is a popular surrogate model-based approach f...

HPC Storage Service Autotuning Using Variational-Autoencoder-Guided Asynchronous Bayesian Optimization

Distributed data storage services tailored to specific applications have...

A portfolio approach to massively parallel Bayesian optimization

One way to reduce the time of conducting optimization studies is to eval...

Federated Bayesian Optimization via Thompson Sampling

Bayesian optimization (BO) is a prominent approach to optimizing expensi...

Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach

Black box optimization requires specifying a search space to explore for...

FastSHAP: Real-Time Shapley Value Estimation

Shapley values are widely used to explain black-box models, but they are...