DeepAI AI Chat
Log In Sign Up

On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks

by   John T. Holodnak, et al.

Multiple techniques for producing calibrated predictive probabilities using deep neural networks in supervised learning settings have emerged that leverage approaches to ensemble diverse solutions discovered during cyclic training or training from multiple random starting points (deep ensembles). However, only a limited amount of work has investigated the utility of exploring the local region around each diverse solution (posterior mode). Using three well-known deep architectures on the CIFAR-10 dataset, we evaluate several simple methods for exploring local regions of the weight space with respect to Brier score, accuracy, and expected calibration error. We consider both Bayesian inference techniques (variational inference and Hamiltonian Monte Carlo applied to the softmax output layer) as well as utilizing the stochastic gradient descent trajectory near optima. While adding separate modes to the ensemble uniformly improves performance, we show that the simple mode exploration methods considered here produce little to no improvement over ensembles without mode exploration.


page 1

page 2

page 3

page 4


Bayesian posterior approximation with stochastic ensembles

We introduce ensembles of stochastic neural networks to approximate the ...

Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift

Bayesian deep learning (BDL) is a promising approach to achieve well-cal...

Deep Ensembles: A Loss Landscape Perspective

Deep ensembles have been empirically shown to be a promising approach fo...

Bayesian Quadrature for Neural Ensemble Search

Ensembling can improve the performance of Neural Networks, but existing ...

What Are Bayesian Neural Network Posteriors Really Like?

The posterior over Bayesian neural network (BNN) parameters is extremely...

Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles

Many practical perception systems exist within larger processes that inc...

Traversing Between Modes in Function Space for Fast Ensembling

Deep ensemble is a simple yet powerful way to improve the performance of...