DeepAI AI Chat
Log In Sign Up

On Stein Variational Neural Network Ensembles

by   Francesco D'Angelo, et al.

Ensembles of deep neural networks have achieved great success recently, but they do not offer a proper Bayesian justification. Moreover, while they allow for averaging of predictions over several hypotheses, they do not provide any guarantees for their diversity, leading to redundant solutions in function space. In contrast, particle-based inference methods, such as Stein variational gradient descent (SVGD), offer a Bayesian framework, but rely on the choice of a kernel to measure the similarity between ensemble members. In this work, we study different SVGD methods operating in the weight space, function space, and in a hybrid setting. We compare the SVGD approaches to other ensembling-based methods in terms of their theoretical properties and assess their empirical performance on synthetic and real-world tasks. We find that SVGD using functional and hybrid kernels can overcome the limitations of deep ensembles. It improves on functional diversity and uncertainty estimation and approaches the true Bayesian posterior more closely. Moreover, we show that using stochastic SVGD updates, as opposed to the standard deterministic ones, can further improve the performance.


page 2

page 15


Repulsive Deep Ensembles are Bayesian

Deep ensembles have recently gained popularity in the deep learning comm...

Efficient variational Bayesian neural network ensembles for outlier detection

In this work we perform outlier detection using ensembles of neural netw...

Calibration and Uncertainty Quantification of Bayesian Convolutional Neural Networks for Geophysical Applications

Deep neural networks offer numerous potential applications across geosci...

Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetries

Deep ensembles (DE) have been successful in improving model performance ...

Packed-Ensembles for Efficient Uncertainty Estimation

Deep Ensembles (DE) are a prominent approach achieving excellent perform...

On the Effectiveness of Mode Exploration in Bayesian Model Averaging for Neural Networks

Multiple techniques for producing calibrated predictive probabilities us...

Greedy Bayesian Posterior Approximation with Deep Ensembles

Ensembles of independently trained neural networks are a state-of-the-ar...