Simple Regularisation for Uncertainty-Aware Knowledge Distillation

05/19/2022
by   Martin Ferianc, et al.
15

Considering uncertainty estimation of modern neural networks (NNs) is one of the most important steps towards deploying machine learning systems to meaningful real-world applications such as in medicine, finance or autonomous systems. At the moment, ensembles of different NNs constitute the state-of-the-art in both accuracy and uncertainty estimation in different tasks. However, ensembles of NNs are unpractical under real-world constraints, since their computation and memory consumption scale linearly with the size of the ensemble, which increase their latency and deployment cost. In this work, we examine a simple regularisation approach for distribution-free knowledge distillation of ensemble of machine learning models into a single NN. The aim of the regularisation is to preserve the diversity, accuracy and uncertainty estimation characteristics of the original ensemble without any intricacies, such as fine-tuning. We demonstrate the generality of the approach on combinations of toy data, SVHN/CIFAR-10, simple to complex NN architectures and different tasks.

READ FULL TEXT

page 1

page 7

page 8

page 10

page 11

research
04/30/2019

Ensemble Distribution Distillation

Ensemble of Neural Network (NN) models are known to yield improvements i...
research
01/14/2020

Hydra: Preserving Ensemble Diversity for Model Distillation

Ensembles of models have been empirically shown to improve predictive pe...
research
03/15/2022

Self-Distribution Distillation: Efficient Uncertainty Estimation

Deep learning is increasingly being applied in safety-critical domains. ...
research
06/26/2022

Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detection

Out-of-distribution (OOD) detection has recently received much attention...
research
10/08/2015

Distilling Model Knowledge

Top-performing machine learning systems, such as deep neural networks, l...
research
09/11/2023

Ensemble-based modeling abstractions for modern self-optimizing systems

In this paper, we extend our ensemble-based component model DEECo with t...
research
08/20/2021

Knowledge Distillation from Ensemble of Offsets for Head Pose Estimation

This paper proposes a method for estimating the head pose from a single ...

Please sign up or login with your details

Forgot password? Click here to reset