Scalable Hyperparameter Optimization with Lazy Gaussian Processes

01/16/2020
by   Raju Ram, et al.
0

Most machine learning methods require careful selection of hyper-parameters in order to train a high performing model with good generalization abilities. Hence, several automatic selection algorithms have been introduced to overcome tedious manual (try and error) tuning of these parameters. Due to its very high sample efficiency, Bayesian Optimization over a Gaussian Processes modeling of the parameter space has become the method of choice. Unfortunately, this approach suffers from a cubic compute complexity due to underlying Cholesky factorization, which makes it very hard to be scaled beyond a small number of sampling steps. In this paper, we present a novel, highly accurate approximation of the underlying Gaussian Process. Reducing its computational complexity from cubic to quadratic allows an efficient strong scaling of Bayesian Optimization while outperforming the previous approach regarding optimization accuracy. The first experiments show speedups of a factor of 162 in single node and further speed up by a factor of 5 in a parallel environment.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/12/2019

Financial Applications of Gaussian Processes and Bayesian Optimization

In the last five years, the financial industry has been impacted by the ...
research
02/27/2019

High-Dimensional Bayesian Optimization with Manifold Gaussian Processes

Bayesian optimization (BO) is a powerful approach for seeking the global...
research
06/13/2012

Practical Bayesian Optimization of Machine Learning Algorithms

Machine learning algorithms frequently require careful tuning of model h...
research
06/19/2020

Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization

Matrix square roots and their inverses arise frequently in machine learn...
research
11/16/2022

Global Optimization with Parametric Function Approximation

We consider the problem of global optimization with noisy zeroth order o...
research
08/22/2019

Adaptive Configuration Oracle for Online Portfolio Selection Methods

Financial markets are complex environments that produce enormous amounts...
research
08/01/2011

Adaptive Gaussian Predictive Process Approximation

We address the issue of knots selection for Gaussian predictive process ...

Please sign up or login with your details

Forgot password? Click here to reset