DeepAI AI Chat
Log In Sign Up

On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks

12/07/2021
by   John T. Holodnak, et al.
0

Multiple techniques for producing calibrated predictive probabilities using deep neural networks in supervised learning settings have emerged that leverage approaches to ensemble diverse solutions discovered during cyclic training or training from multiple random starting points (deep ensembles). However, only a limited amount of work has investigated the utility of exploring the local region around each diverse solution (posterior mode). Using three well-known deep architectures on the CIFAR-10 dataset, we evaluate several simple methods for exploring local regions of the weight space with respect to Brier score, accuracy, and expected calibration error. We consider both Bayesian inference techniques (variational inference and Hamiltonian Monte Carlo applied to the softmax output layer) as well as utilizing the stochastic gradient descent trajectory near optima. While adding separate modes to the ensemble uniformly improves performance, we show that the simple mode exploration methods considered here produce little to no improvement over ensembles without mode exploration.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/15/2022

Bayesian posterior approximation with stochastic ensembles

We introduce ensembles of stochastic neural networks to approximate the ...
06/21/2023

Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift

Bayesian deep learning (BDL) is a promising approach to achieve well-cal...
12/05/2019

Deep Ensembles: A Loss Landscape Perspective

Deep ensembles have been empirically shown to be a promising approach fo...
03/15/2023

Bayesian Quadrature for Neural Ensemble Search

Ensembling can improve the performance of Neural Networks, but existing ...
04/29/2021

What Are Bayesian Neural Network Posteriors Really Like?

The posterior over Bayesian neural network (BNN) parameters is extremely...
06/24/2016

Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles

Many practical perception systems exist within larger processes that inc...
06/20/2023

Traversing Between Modes in Function Space for Fast Ensembling

Deep ensemble is a simple yet powerful way to improve the performance of...