Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models

03/24/2018
by   Hugh Salimbeni, et al.
0

The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning. We conclude that the natural gradient can significantly improve performance in terms of wall-clock time. For ill-conditioned posteriors the benefit of the natural gradient method is especially pronounced, and we demonstrate a practical setting where ordinary gradients are unusable. We show how natural gradients can be computed efficiently and automatically in any parameterization, using automatic differentiation. Our code is integrated into the GPflow package.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

11/05/2021

Dual Parameterization of Sparse Variational Gaussian Processes

Sparse variational Gaussian process (SVGP) methods are a common choice f...
11/22/2019

A Fully Natural Gradient Scheme for Improving Inference of the Heterogeneous Multi-Output Gaussian Process Model

A recent novel extension of multi-output Gaussian processes handles hete...
03/07/2019

The Variational Predictive Natural Gradient

Variational inference transforms posterior inference into parametric opt...
06/13/2014

Smoothed Gradients for Stochastic Variational Inference

Stochastic variational inference (SVI) lets us scale up Bayesian computa...
10/27/2016

GPflow: A Gaussian process library using TensorFlow

GPflow is a Gaussian process library that uses TensorFlow for its core c...
05/08/2012

The Natural Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its Use

The natural gradient allows for more efficient gradient descent by remov...
08/03/2020

Multifidelity Data Fusion via Gradient-Enhanced Gaussian Process Regression

We propose a data fusion method based on multi-fidelity Gaussian process...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.