Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence

by   Konstantin Berestizshevsky, et al.

We study the tradeoff between computational effort and accuracy in a cascade of deep neural networks. During inference, early termination in the cascade is controlled by confidence levels derived directly from the softmax outputs of intermediate classifiers. The advantage of early termination is that classification is performed using less computation, thus adjusting the computational effort to the complexity of the input. Moreover, dynamic modification of confidence thresholds allow one to trade accuracy for computational effort without requiring retraining. Basing of early termination on softmax classifier outputs is justified by experimentation that demonstrates an almost linear relation between confidence levels in intermediate classifiers and accuracy. Our experimentation with architectures based on ResNet obtained the following results. (i) A speedup of 1.5 that sacrifices 1.4 respect to the CIFAR-10 test set. (ii) A speedup of 1.19 that sacrifices 0.7 accuracy with respect to the CIFAR-100 test set. (iii) A speedup of 2.16 that sacrifices 1.4


page 1

page 2

page 3

page 4


BasisNet: Two-stage Model Synthesis for Efficient Inference

In this work, we present BasisNet which combines recent advancements in ...

Real-Time Background Subtraction Using Adaptive Sampling and Cascade of Gaussians

Background-Foreground classification is a fundamental well-studied probl...

Resource-Constrained Edge AI with Early Exit Prediction

By leveraging the data sample diversity, the early-exit network recently...

Overinterpretation reveals image classification model pathologies

Image classifiers are typically scored on their test set accuracy, but h...

Confidence estimation in Deep Neural networks via density modelling

State-of-the-art Deep Neural Networks can be easily fooled into providin...

Improving Simple Models with Confidence Profiles

In this paper, we propose a new method called ProfWeight for transferrin...

Training for Fast Sequential Prediction Using Dynamic Feature Selection

We present paired learning and inference algorithms for significantly re...