Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling

06/15/2022
by   Jiri Hron, et al.
0

We introduce repriorisation, a data-dependent reparameterisation which transforms a Bayesian neural network (BNN) posterior to a distribution whose KL divergence to the BNN prior vanishes as layer widths grow. The repriorisation map acts directly on parameters, and its analytic simplicity complements the known neural network Gaussian process (NNGP) behaviour of wide BNNs in function space. Exploiting the repriorisation, we develop a Markov chain Monte Carlo (MCMC) posterior sampling algorithm which mixes faster the wider the BNN. This contrasts with the typically poor performance of MCMC in high dimensions. We observe up to 50x higher effective sample size relative to no reparametrisation for both fully-connected and residual networks. Improvements are achieved at all widths, with the margin between reparametrised and standard BNNs growing with layer width.

READ FULL TEXT

page 2

page 5

research
07/16/2021

Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods

Normalizing flows can generate complex target distributions and thus sho...
research
05/02/2021

Bayesian structure learning and sampling of Bayesian networks with the R package BiDAG

The R package BiDAG implements Markov chain Monte Carlo (MCMC) methods f...
research
01/18/2022

Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

A fundamental challenge in Bayesian inference is efficient representatio...
research
09/28/2020

Quantile Regression Neural Networks: A Bayesian Approach

This article introduces a Bayesian neural network estimation method for ...
research
06/08/2020

Efficient MCMC Sampling for Bayesian Matrix Factorization by Breaking Posterior Symmetries

Bayesian low-rank matrix factorization techniques have become an essenti...
research
05/28/2020

Bayesian Restoration of Audio Degraded by Low-Frequency Pulses Modeled via Gaussian Process

A common defect found when reproducing old vinyl and gramophone recordin...
research
03/09/2020

MCMC Guided CNN Training and Segmentation for Pancreas Extraction

Efficient organ segmentation is the precondition of various quantitative...

Please sign up or login with your details

Forgot password? Click here to reset