DeepAI AI Chat
Log In Sign Up

Mixed Variable Bayesian Optimization with Frequency Modulated Kernels

02/25/2021
by   ChangYong Oh, et al.
13

The sample efficiency of Bayesian optimization(BO) is often boosted by Gaussian Process(GP) surrogate models. However, on mixed variable spaces, surrogate models other than GPs are prevalent, mainly due to the lack of kernels which can model complex dependencies across different types of variables. In this paper, we propose the frequency modulated (FM) kernel flexibly modeling dependencies among different types of variables, so that BO can enjoy the further improved sample efficiency. The FM kernel uses distances on continuous variables to modulate the graph Fourier spectrum derived from discrete variables. However, the frequency modulation does not always define a kernel with the similarity measure behavior which returns higher values for pairs of more similar points. Therefore, we specify and prove conditions for FM kernels to be positive definite and to exhibit the similarity measure behavior. In experiments, we demonstrate the improved sample efficiency of GP BO using FM kernels (BO-FM).On synthetic problems and hyperparameter optimization problems, BO-FM outperforms competitors consistently. Also, the importance of the frequency modulation principle is empirically demonstrated on the same problems. On joint optimization of neural architectures and SGD hyperparameters, BO-FM outperforms competitors including Regularized evolution(RE) and BOHB. Remarkably, BO-FM performs better even than RE and BOHB using three times as many evaluations.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/01/2019

Combinatorial Bayesian Optimization using Graph Representations

This paper focuses on Bayesian Optimization - typically considered with ...
11/15/2022

A mixed-categorical correlation kernel for Gaussian process

Recently, there has been a growing interest for mixed-categorical meta-m...
05/23/2019

Bayesian Optimization over Sets

We propose a Bayesian optimization method over sets, to minimize a black...
10/03/2019

Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables

Although Bayesian Optimization (BO) has been employed for accelerating m...
10/02/2020

BOSS: Bayesian Optimization over String Spaces

This article develops a Bayesian optimization (BO) method which acts dir...
06/08/2021

Bayesian Optimization over Hybrid Spaces

We consider the problem of optimizing hybrid structures (mixture of disc...
06/05/2017

Ensemble Bayesian Optimization

Bayesian Optimization (BO) has been shown to be a very effective paradig...