Neural Collapse with Cross-Entropy Loss

12/15/2020

∙

We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in ℝ^d. We prove that when d ≥ n - 1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also show a connection with the frame potential of Benedetto Fickus.

READ FULL TEXT

Neural Collapse with Cross-Entropy Loss

Taming the Cross Entropy Loss

Multi-Frame Cross-Entropy Training for Convolutional Neural Networks in Speech Recognition

Adaptively Solving the Local-Minimum Problem for Deep Neural Networks

The Tree Loss: Improving Generalization with Many Classes

Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference

Cross entropy as objective function for music generative models

Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap

Neural Collapse with Cross-Entropy Loss

Related Research

Taming the Cross Entropy Loss

Multi-Frame Cross-Entropy Training for Convolutional Neural Networks in Speech Recognition

Adaptively Solving the Local-Minimum Problem for Deep Neural Networks

The Tree Loss: Improving Generalization with Many Classes

Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference

Cross entropy as objective function for music generative models

Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap