Variational Orthogonal Features

06/23/2020

∙

Sparse stochastic variational inference allows Gaussian process models to be applied to large datasets. The per iteration computational cost of inference with this method is 𝒪(ÑM^2+M^3), where Ñ is the number of points in a minibatch and M is the number of `inducing features', which determine the expressiveness of the variational family. Several recent works have shown that for certain priors, features can be defined that remove the 𝒪(M^3) cost of computing a minibatch estimate of an evidence lower bound (ELBO). This represents a significant computational savings when M≫Ñ. We present a construction of features for any stationary prior kernel that allow for computation of an unbiased estimator to the ELBO using T Monte Carlo samples in 𝒪(ÑT+M^2T) and in 𝒪(ÑT+MT) with an additional approximation. We analyze the impact of this additional approximation on inference quality.

READ FULL TEXT

Variational Orthogonal Features

Unbiased Implicit Variational Inference

Quadruply Stochastic Gaussian Processes

Stochastic Variational Inference for Fully Bayesian Sparse Gaussian Process Regression Models

Quantized Variational Inference

Generic Inference in Latent Gaussian Process Models

Rates of Convergence for Sparse Variational Gaussian Process Regression

Sparse Variational Inference: Bayesian Coresets from Scratch

Variational Orthogonal Features

Related Research

Unbiased Implicit Variational Inference

Quadruply Stochastic Gaussian Processes

Stochastic Variational Inference for Fully Bayesian Sparse Gaussian Process Regression Models

Quantized Variational Inference

Generic Inference in Latent Gaussian Process Models

Rates of Convergence for Sparse Variational Gaussian Process Regression

Sparse Variational Inference: Bayesian Coresets from Scratch