What Are Bayesian Neural Network Posteriors Really Like?

by   Pavel Izmailov, et al.

The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using inexpensive mini-batch methods such as mean-field variational inference or stochastic-gradient Markov chain Monte Carlo (SGMCMC). To investigate foundational questions in Bayesian deep learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern architectures. We show that (1) BNNs can achieve significant performance gains over standard training and deep ensembles; (2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains; (3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a "cold posterior" effect, which we show is largely an artifact of data augmentation; (4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5) Bayesian neural networks show surprisingly poor generalization under domain shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC methods can provide good generalization, they provide distinct predictive distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.


page 6

page 20


Bayesian posterior approximation with stochastic ensembles

We introduce ensembles of stochastic neural networks to approximate the ...

Learning to solve Bayesian inverse problems: An amortized variational inference approach

Inverse problems, i.e., estimating parameters of physical models from ex...

Dangers of Bayesian Model Averaging under Covariate Shift

Approximate Bayesian inference for neural networks is considered a robus...

Practical Deep Learning with Bayesian Principles

Bayesian methods promise to fix many shortcomings of deep learning, but ...

Variational Bayes Deep Operator Network: A data-driven Bayesian solver for parametric differential equations

Neural network based data-driven operator learning schemes have shown tr...

Rate Optimal Variational Bayesian Inference for Sparse DNN

Sparse deep neural network (DNN) has drawn much attention in recent stud...

On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks

Multiple techniques for producing calibrated predictive probabilities us...

Please sign up or login with your details

Forgot password? Click here to reset