Procrustean Training for Imbalanced Deep Learning

04/05/2021
by   Han-Jia Ye, et al.
0

Neural networks trained with class-imbalanced data are known to perform poorly on minor classes of scarce training data. Several recent works attribute this to over-fitting to minor classes. In this paper, we provide a novel explanation of this issue. We found that a neural network tends to first under-fit the minor classes by classifying most of their data into the major classes in early training epochs. To correct these wrong predictions, the neural network then must focus on pushing features of minor class data across the decision boundaries between major and minor classes, leading to much larger gradients for features of minor classes. We argue that such an under-fitting phase over-emphasizes the competition between major and minor classes, hinders the neural network from learning the discriminative knowledge that can be generalized to test data, and eventually results in over-fitting. To address this issue, we propose a novel learning strategy to equalize the training progress across classes. We mix features of the major class data with those of other data in a mini-batch, intentionally weakening their features to prevent a neural network from fitting them first. We show that this strategy can largely balance the training accuracy and feature gradients across classes, effectively mitigating the under-fitting then over-fitting problem for minor class data. On several benchmark datasets, our approach achieves the state-of-the-art accuracy, especially for the challenging step-imbalanced cases.

READ FULL TEXT
01/06/2020

Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning

We investigate learning a ConvNet classifier with class-imbalanced data....
04/01/2020

M2m: Imbalanced Classification via Major-to-minor Translation

In most real-world scenarios, labeled training datasets are highly class...
04/19/2022

Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning

Class imbalance distribution widely exists in real-world engineering. Ho...
07/17/2018

Pseudo-Feature Generation for Imbalanced Data Analysis in Deep Learning

We generate pseudo-features by multivariate probability distributions ob...
07/12/2017

Influence of Resampling on Accuracy of Imbalanced Classification

In many real-world binary classification tasks (e.g. detection of certai...
02/09/2018

Deep Learning for Malicious Flow Detection

Cyber security has grown up to be a hot issue in recent years. How to id...
12/16/2021

Sensitivity Analysis of the MCRF Model to Different Transiogram Joint Modeling Methods for Simulating Categorical Spatial Variables

Markov chain geostatistics is a methodology for simulating categorical f...