Bayesian Deep Learning and a Probabilistic Perspective of Generalization

by   Andrew Gordon Wilson, et al.

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.


page 15

page 24

page 25

page 26


The Case for Bayesian Deep Learning

The key distinguishing property of a Bayesian approach is marginalizatio...

A Probabilistic Representation of Deep Learning

In this work, we introduce a novel probabilistic representation of deep ...

Probabilistic Meta-Representations Of Neural Networks

Existing Bayesian treatments of neural networks are typically characteri...

Neuro-symbolic Neurodegenerative Disease Modeling as Probabilistic Programmed Deep Kernels

We present a probabilistic programmed deep kernel learning approach to p...

Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective

The lottery ticket hypothesis (LTH) has attracted attention because it c...

Model Generalization: A Sharpness Aware Optimization Perspective

Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimiza...

Correcting Model Bias with Sparse Implicit Processes

Model selection in machine learning (ML) is a crucial part of the Bayesi...

Code Repositories