DeepAI AI Chat
Log In Sign Up

Greedy Bayesian Posterior Approximation with Deep Ensembles

by   Aleksei Tiulpin, et al.

Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixture of delta functions. The training of ensembles relies on non-convexity of the loss landscape and random initialization of their individual members, making the resulting posterior approximation uncontrolled. This paper proposes a novel and principled method to tackle this limitation, minimizing an f-divergence between the true posterior and a kernel density estimator in a function space. We analyze this objective from a combinatorial point of view, and show that it is submodular with respect to mixture components for any f. Subsequently, we consider the problem of greedy ensemble construction, and from the marginal gain of the total objective, we derive a novel diversity term for ensemble methods. The performance of our approach is demonstrated on computer vision out-of-distribution benchmarks in a range of architectures trained on multiple datasets. The source code of our method is publicly available at


page 1

page 2

page 3

page 4


Repulsive Deep Ensembles are Bayesian

Deep ensembles have recently gained popularity in the deep learning comm...

SAE: Sequential Anchored Ensembles

Computing the Bayesian posterior of a neural network is a challenging ta...

Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification

We consider the problem of uncertainty quantification in high dimensiona...

Bayesian Neural Network Ensembles

Ensembles of neural networks (NNs) have long been used to estimate predi...

Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles

Understanding and quantifying uncertainty in black box Neural Networks (...

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

Deep ensembles perform better than a single network thanks to the divers...

Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity

While deep neural networks show great performance on fitting to the trai...

Code Repositories


Greedy Bayesian Posterior Approximation with Deep Ensembles. A. Tiulpin and M. B. Blaschko. (2021)

view repo