DeepAI AI Chat
Log In Sign Up

SAE: Sequential Anchored Ensembles

by   Arnaud Delaunoy, et al.
University of Liège

Computing the Bayesian posterior of a neural network is a challenging task due to the high-dimensionality of the parameter space. Anchored ensembles approximate the posterior by training an ensemble of neural networks on anchored losses designed for the optima to follow the Bayesian posterior. Training an ensemble, however, becomes computationally expensive as its number of members grows since the full training procedure is repeated for each member. In this note, we present Sequential Anchored Ensembles (SAE), a lightweight alternative to anchored ensembles. Instead of training each member of the ensemble from scratch, the members are trained sequentially on losses sampled with high auto-correlation, hence enabling fast convergence of the neural networks and efficient approximation of the Bayesian posterior. SAE outperform anchored ensembles, for a given computational budget, on some benchmarks while showing comparable performance on the others and achieved 2nd and 3rd place in the light and extended tracks of the NeurIPS 2021 Approximate Inference in Bayesian Deep Learning competition.


page 1

page 2

page 3

page 4


Repulsive Deep Ensembles are Bayesian

Deep ensembles have recently gained popularity in the deep learning comm...

Bayesian Deep Ensembles via the Neural Tangent Kernel

We explore the link between deep ensembles and Gaussian processes (GPs) ...

Greedy Bayesian Posterior Approximation with Deep Ensembles

Ensembles of independently trained neural networks are a state-of-the-ar...

Bayesian Neural Network Ensembles

Ensembles of neural networks (NNs) have long been used to estimate predi...

Hypermodels for Exploration

We study the use of hypermodels to represent epistemic uncertainty and g...

Neural network ensembles: Evaluation of aggregation algorithms

Ensembles of artificial neural networks show improved generalization cap...