DeepAI AI Chat
Log In Sign Up

Greedy Bayesian Posterior Approximation with Deep Ensembles

05/29/2021
by   Aleksei Tiulpin, et al.
0

Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixture of delta functions. The training of ensembles relies on non-convexity of the loss landscape and random initialization of their individual members, making the resulting posterior approximation uncontrolled. This paper proposes a novel and principled method to tackle this limitation, minimizing an f-divergence between the true posterior and a kernel density estimator in a function space. We analyze this objective from a combinatorial point of view, and show that it is submodular with respect to mixture components for any f. Subsequently, we consider the problem of greedy ensemble construction, and from the marginal gain of the total objective, we derive a novel diversity term for ensemble methods. The performance of our approach is demonstrated on computer vision out-of-distribution benchmarks in a range of architectures trained on multiple datasets. The source code of our method is publicly available at https://github.com/MIPT-Oulu/greedy_ensembles_training.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/22/2021

Repulsive Deep Ensembles are Bayesian

Deep ensembles have recently gained popularity in the deep learning comm...
12/30/2021

SAE: Sequential Anchored Ensembles

Computing the Bayesian posterior of a neural network is a challenging ta...
04/08/2023

Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification

We consider the problem of uncertainty quantification in high dimensiona...
11/27/2018

Bayesian Neural Network Ensembles

Ensembles of neural networks (NNs) have long been used to estimate predi...
09/25/2020

Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles

Understanding and quantifying uncertainty in black box Neural Networks (...
01/14/2021

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

Deep ensembles perform better than a single network thanks to the divers...
02/05/2021

Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity

While deep neural networks show great performance on fitting to the trai...

Code Repositories

greedy_ensembles_training

Greedy Bayesian Posterior Approximation with Deep Ensembles. A. Tiulpin and M. B. Blaschko. (2021)


view repo