IKD+: Reliable Low Complexity Deep Models For Retinopathy Classification

03/04/2023
by   Shreyas Bhat Brahmavar, et al.
0

Deep neural network (DNN) models for retinopathy have estimated predictive accuracies in the mid-to-high 90 unaddressed: State-of-the-art models are complex and require substantial computational infrastructure to train and deploy; The reliability of predictions can vary widely. In this paper, we focus on these aspects and propose a form of iterative knowledge distillation(IKD), called IKD+ that incorporates a tradeoff between size, accuracy and reliability. We investigate the functioning of IKD+ using two widely used techniques for estimating model calibration (Platt-scaling and temperature-scaling), using the best-performing model available, which is an ensemble of EfficientNets with approximately 100M parameters. We demonstrate that IKD+ equipped with temperature-scaling results in models that show up to approximately 500-fold decreases in the number of parameters than the original ensemble without a significant loss in accuracy. In addition, calibration scores (reliability) for the IKD+ models are as good as or better than the base mode

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/24/2023

Rethinking Data Distillation: Do Not Overlook Calibration

Neural networks trained on distilled data often produce over-confident o...
research
05/01/2019

Unsupervised Temperature Scaling: Post-Processing Unsupervised Calibration of Deep Models Decisions

Great performances of deep learning are undeniable, with impressive resu...
research
11/18/2022

Layer-Stack Temperature Scaling

Recent works demonstrate that early layers in a neural network contain u...
research
11/09/2020

Knowledge Distillation for Singing Voice Detection

Singing Voice Detection (SVD) has been an active area of research in mus...
research
11/11/2022

PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation

Pre-trained language models have become a crucial part of ranking system...
research
07/02/2018

weight-importance sparse training in keyword spotting

Large size models are implemented in recently ASR system to deal with co...
research
01/02/2021

Uncertainty-sensitive Activity Recognition: a Reliability Benchmark and the CARING Models

Beyond assigning the correct class, an activity recognition model should...

Please sign up or login with your details

Forgot password? Click here to reset