DeepAI AI Chat
Log In Sign Up

Packed-Ensembles for Efficient Uncertainty Estimation

10/17/2022
by   Olivier Laurent, et al.
15

Deep Ensembles (DE) are a prominent approach achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world systems constrain to smaller ensembles and lower capacity networks, significantly deteriorating their performance and properties. We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolutions to parallelize the ensemble into a single common backbone and forward pass to improve training and inference speeds. PE is designed to work under the memory budget of a single standard neural network. Through extensive studies we show that PE faithfully preserve the properties of DE, e.g., diversity, and match their performance in terms of accuracy, calibration, out-of-distribution detection and robustness to distribution shift.

READ FULL TEXT
01/26/2022

Improving robustness and calibration in ensembles with diversity regularization

Calibration and uncertainty estimation are crucial topics in high-risk e...
02/14/2022

Deep Ensembles Work, But Are They Necessary?

Ensembling neural networks is an effective way to increase accuracy, and...
06/01/2022

Sequential Bayesian Neural Subnetwork Ensembles

Deep neural network ensembles that appeal to model diversity have been u...
05/01/2020

When Ensembling Smaller Models is More Efficient than Single Large Models

Ensembling is a simple and popular technique for boosting evaluation per...
10/10/2022

Layer Ensembles

Deep Ensembles, as a type of Bayesian Neural Networks, can be used to es...
10/07/2021

Sparse MoEs meet Efficient Ensembles

Machine learning models based on the aggregated outputs of submodels, ei...
02/22/2018

Diversity regularization in deep ensembles

Calibrating the confidence of supervised learning models is important fo...