Gaussian Mean Field Regularizes by Limiting Learned Information

02/12/2019
by   Julius Kunze, et al.
0

Variational inference with a factorized Gaussian posterior estimate is a widely used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is rescaled. Our experiments demonstrate that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.

READ FULL TEXT
research
06/13/2021

Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data

Variational inference enables approximate posterior inference of the hig...
research
09/15/2022

On the detrimental effect of invariances in the likelihood for variational inference

Variational Bayesian posterior inference often requires simplifying appr...
research
04/16/2014

Structured Stochastic Variational Inference

Stochastic variational inference makes it possible to approximate poster...
research
02/10/2020

Try Depth Instead of Weight Correlations: Mean-field is a Less Restrictive Assumption for Deeper Networks

We challenge the longstanding assumption that the mean-field approximati...
research
07/08/2022

Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical study

This paper studies the Variational Inference (VI) used for training Baye...
research
02/07/2019

Radial and Directional Posteriors for Bayesian Neural Networks

We propose a new variational family for Bayesian neural networks. We dec...
research
07/13/2020

Model Fusion with Kullback–Leibler Divergence

We propose a method to fuse posterior distributions learned from heterog...

Please sign up or login with your details

Forgot password? Click here to reset