Log In Sign Up

Instance-based Label Smoothing For Better Calibrated Classification Networks

by   Mohamed Maher, et al.

Label smoothing is widely used in deep neural networks for multi-class classification. While it enhances model generalization and reduces overconfidence by aiming to lower the probability for the predicted class, it distorts the predicted probabilities of other classes resulting in poor class-wise calibration. Another method for enhancing model generalization is self-distillation where the predictions of a teacher network trained with one-hot labels are used as the target for training a student network. We take inspiration from both label smoothing and self-distillation and propose two novel instance-based label smoothing approaches, where a teacher network trained with hard one-hot labels is used to determine the amount of per class smoothness applied to each instance. The assigned smoothing factor is non-uniformly distributed along with the classes according to their similarity with the actual class. Our methods show better generalization and calibration over standard label smoothing on various deep neural architectures and image classification datasets.


When Does Label Smoothing Help?

The generalization and learning speed of a multi-class neural network ca...

Self-Distillation as Instance-Specific Label Smoothing

It has been recently demonstrated that multi-generational self-distillat...

Class-Similarity Based Label Smoothing for Generalized Confidence Calibration

Since modern neural networks are known to be overconfident, several tech...

Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation

Overconfidence has been shown to impair generalization and calibration o...

Confidence-Aware Calibration and Scoring Functions for Curriculum Learning

Despite the great success of state-of-the-art deep neural networks, seve...

On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

Mixup zhang2017mixup is a recently proposed method for training deep neu...

Calibrating Histopathology Image Classifiers using Label Smoothing

The classification of histopathology images fundamentally differs from t...