Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees

11/02/2021
by   William J. Wilkinson, et al.
0

We formulate natural gradient variational inference (VI), expectation propagation (EP), and posterior linearisation (PL) as extensions of Newton's method for optimising the parameters of a Bayesian posterior distribution. This viewpoint explicitly casts inference algorithms under the framework of numerical optimisation. We show that common approximations to Newton's method from the optimisation literature, namely Gauss-Newton and quasi-Newton methods (e.g., the BFGS algorithm), are still valid under this 'Bayes-Newton' framework. This leads to a suite of novel algorithms which are guaranteed to result in positive semi-definite covariance matrices, unlike standard VI and EP. Our unifying viewpoint provides new insights into the connections between various inference schemes. All the presented methods apply to any model with a Gaussian prior and non-conjugate likelihood, which we demonstrate with (sparse) Gaussian processes and state space models.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/21/2020

Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks

In this thesis, we disentangle the generalized Gauss-Newton and approxim...
07/12/2020

State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes

We formulate approximate Bayesian inference in non-conjugate temporal an...
02/27/2019

Quasi-Bayes properties of a recursive procedure for mixtures

Bayesian methods are attractive and often optimal, yet nowadays pressure...
03/04/2011

Multiple Kernel Learning: A Unifying Probabilistic Viewpoint

We present a probabilistic viewpoint to multiple kernel learning unifyin...
02/24/2020

Handling the Positive-Definite Constraint in the Bayesian Learning Rule

Bayesian learning rule is a recently proposed variational inference meth...
04/03/2017

Efficient acquisition rules for model-based approximate Bayesian computation

Approximate Bayesian computation (ABC) is a method for Bayesian inferenc...
02/10/2014

Probabilistic Interpretation of Linear Solvers

This manuscript proposes a probabilistic framework for algorithms that i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.