DeepAI AI Chat
Log In Sign Up

Combining Bayesian Optimization and Lipschitz Optimization

10/10/2018
by   Mohamed Osama Ahmed, et al.
The University of British Columbia
0

Bayesian optimization and Lipschitz optimization have developed alternative techniques for optimizing black-box functions. They each exploit a different form of prior about the function. In this work, we explore strategies to combine these techniques for better global optimization. In particular, we propose ways to use the Lipschitz continuity assumption within traditional BO algorithms, which we call Lipschitz Bayesian optimization (LBO). This approach does not increase the asymptotic runtime and in some cases drastically improves the performance (while in the worst case the performance is similar). Indeed, in a particular setting, we prove that using the Lipschitz information yields the same or a better bound on the regret compared to using Bayesian optimization on its own. Moreover, we propose a simple heuristics to estimate the Lipschitz constant, and prove that a growing estimate of the Lipschitz constant is in some sense "harmless". Our experiments on 15 datasets with 4 acquisition functions show that in the worst case LBO performs similar to the underlying BO method while in some cases it performs substantially better. Thompson sampling in particular typically saw drastic improvements (as the Lipschitz information corrected for it's well-known "over-exploration" phenomenon) and its LBO variant often outperformed other acquisition functions.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/08/2023

Bayesian Optimization of Expensive Nested Grey-Box Functions

We consider the problem of optimizing a grey-box objective function, i.e...
07/29/2021

Bayesian Optimization for Min Max Optimization

A solution that is only reliable under favourable conditions is hardly a...
03/30/2012

A Lipschitz Exploration-Exploitation Scheme for Bayesian Optimization

The problem of optimizing unknown costly-to-evaluate functions has been ...
04/30/2020

The Lipschitz Constant of Perturbed Anonymous Games

The worst-case Lipschitz constant of an n-player k-action δ-perturbed ga...
03/30/2016

Robustness of Bayesian Pool-based Active Learning Against Prior Misspecification

We study the robustness of active learning (AL) algorithms against prior...
04/30/2021

ModelGuard: Runtime Validation of Lipschitz-continuous Models

This paper presents ModelGuard, a sampling-based approach to runtime mod...