Neural Network Classifier as Mutual Information Evaluator

06/19/2021
by   Zhenyue Qin, et al.
10

Cross-entropy loss with softmax output is a standard choice to train neural network classifiers. We give a new view of neural network classifiers with softmax and cross-entropy as mutual information evaluators. We show that when the dataset is balanced, training a neural network with cross-entropy maximises the mutual information between inputs and labels through a variational form of mutual information. Thereby, we develop a new form of softmax that also converts a classifier to a mutual information evaluator when the dataset is imbalanced. Experimental results show that the new form leads to better classification accuracy, in particular for imbalanced datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/25/2019

Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator

Mutual information is widely applied to learn latent representations of ...
research
09/21/2022

Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems

Deep learning systems have been reported to achieve state-of-the-art per...
research
11/10/2018

Formal Limitations on the Measurement of Mutual Information

Motivate by applications to unsupervised learning, we consider the probl...
research
05/28/2019

Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution

Machine learning theory has mostly focused on generalization to samples ...
research
07/16/2020

Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement

Cross Entropy (CE) has an important role in machine learning and, in par...
research
03/29/2018

Modified SMOTE Using Mutual Information and Different Sorts of Entropies

SMOTE is one of the oversampling techniques for balancing the datasets a...
research
01/26/2018

Weakly Supervised Object Detection with Pointwise Mutual Information

In this work a novel approach for weakly supervised object detection tha...

Please sign up or login with your details

Forgot password? Click here to reset