Variational Laplace for Bayesian neural networks

02/27/2021
by   Ali Unlu, et al.
0

We develop variational Laplace for Bayesian neural networks (BNNs) which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. Variational Laplace performs better on image classification tasks than MAP inference and far better than standard variational inference with stochastic sampling despite using the same mean-field Gaussian approximate posterior. The Variational Laplace objective is simple to evaluate, as it is (in essence) the log-likelihood, plus weight-decay, plus a squared-gradient regularizer. Finally, we emphasise care needed in benchmarking standard VI as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

11/20/2020

Gradient Regularisation as Approximate Variational Inference

Variational inference in Bayesian neural networks is usually performed u...
01/15/2019

Mixed Variational Inference

The Laplace approximation has been one of the workhorses of Bayesian inf...
07/21/2020

Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks

In this thesis, we disentangle the generalized Gauss-Newton and approxim...
02/07/2020

The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Variational Bayesian Inference is a popular methodology for approximatin...
06/10/2019

Stochastic Neural Network with Kronecker Flow

Recent advances in variational inference enable the modelling of highly ...
06/13/2018

Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

Uncertainty computation in deep learning is essential to design robust a...
04/08/2019

Bayesian Neural Networks at Finite Temperature

We recapitulate the Bayesian formulation of neural network based classif...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.