A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression

by   Quang Minh Hoang, et al.

While much research effort has been dedicated to scaling up sparse Gaussian process (GP) models based on inducing variables for big data, little attention is afforded to the other less explored class of low-rank GP approximations that exploit the sparse spectral representation of a GP kernel. This paper presents such an effort to advance the state of the art of sparse spectrum GP models to achieve competitive predictive performance for massive datasets. Our generalized framework of stochastic variational Bayesian sparse spectrum GP (sVBSSGP) models addresses their shortcomings by adopting a Bayesian treatment of the spectral frequencies to avoid overfitting, modeling these frequencies jointly in its variational distribution to enable their interaction a posteriori, and exploiting local data for boosting the predictive performance. However, such structural improvements result in a variational lower bound that is intractable to be optimized. To resolve this, we exploit a variational parameterization trick to make it amenable to stochastic optimization. Interestingly, the resulting stochastic gradient has a linearly decomposable structure that can be exploited to refine our stochastic optimization method to incur constant time per iteration while preserving its property of being an unbiased estimator of the exact gradient of the variational lower bound. Empirical evaluation on real-world datasets shows that sVBSSGP outperforms state-of-the-art stochastic implementations of sparse GP models.


page 1

page 2

page 3

page 4


Stochastic Variational Inference for Fully Bayesian Sparse Gaussian Process Regression Models

This paper presents a novel variational inference framework for deriving...

Scalable Variational Bayesian Kernel Selection for Sparse Gaussian Process Regression

This paper presents a variational Bayesian kernel selection (VBKS) algor...

Parallel Gaussian Process Regression for Big Data: Low-Rank Representation Meets Markov Approximation

The expressive power of a Gaussian process (GP) model comes at a cost of...

Sparse Spectrum Gaussian Process for Bayesian Optimisation

We propose a novel sparse spectrum approximation of Gaussian process (GP...

Generic Inference in Latent Gaussian Process Models

We develop an automated variational method for inference in models with ...

Continual Multi-task Gaussian Processes

We address the problem of continual learning in multi-task Gaussian proc...

Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation

We propose an efficient stochastic variational approach to GP classifica...

Please sign up or login with your details

Forgot password? Click here to reset