Wide Mean-Field Bayesian Neural Networks Ignore the Data

02/23/2022
by   Beau Coker, et al.
0

Bayesian neural networks (BNNs) combine the expressive power of deep learning with the advantages of Bayesian formalism. In recent years, the analysis of wide, deep BNNs has provided theoretical insight into their priors and posteriors. However, we have no analogous insight into their posteriors under approximate inference. In this work, we show that mean-field variational inference entirely fails to model the data when the network width is large and the activation function is odd. Specifically, for fully-connected BNNs with odd activation functions and a homoscedastic Gaussian likelihood, we show that the optimal mean-field variational posterior predictive (i.e., function space) distribution converges to the prior predictive distribution as the width tends to infinity. We generalize aspects of this result to other likelihoods. Our theoretical results are suggestive of underfitting behavior previously observered in BNNs. While our convergence bounds are non-asymptotic and constants in our analysis can be computed, they are currently too loose to be applicable in standard training regimes. Finally, we show that the optimal approximate posterior need not tend to the prior if the activation function is not odd, showing that our statements cannot be generalized arbitrarily.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/13/2021

Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data

Variational inference enables approximate posterior inference of the hig...
research
07/26/2022

One Simple Trick to Fix Your Bayesian Neural Network

One of the most popular estimation methods in Bayesian neural networks (...
research
09/15/2022

On the detrimental effect of invariances in the likelihood for variational inference

Variational Bayesian posterior inference often requires simplifying appr...
research
07/08/2022

Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study

This paper studies the Variational Inference (VI) used for training Baye...
research
02/07/2020

The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Variational Bayesian Inference is a popular methodology for approximatin...
research
07/01/2019

Radial Bayesian Neural Networks: Robust Variational Inference In Big Models

We propose Radial Bayesian Neural Networks: a variational distribution f...
research
11/29/2019

Richer priors for infinitely wide multi-layer perceptrons

It is well-known that the distribution over functions induced through a ...

Please sign up or login with your details

Forgot password? Click here to reset