Perturbation Analysis of Neural Collapse

10/29/2022
by   Tom Tirer, et al.
0

Training deep neural networks for classification often includes minimizing the training loss beyond the zero training error point. In this phase of training, a "neural collapse" behavior has been observed: the variability of features (outputs of the penultimate layer) of within-class samples decreases and the mean features of different classes approach a certain tight frame structure. Recent works analyze this behavior via idealized unconstrained features models where all the minimizers exhibit exact collapse. However, with practical networks and datasets, the features typically do not reach exact collapse, e.g., because deep layers cannot arbitrarily modify intermediate features that are far from being collapsed. In this paper, we propose a richer model that can capture this phenomenon by forcing the features to stay in the vicinity of a predefined features matrix (e.g., intermediate features). We explore the model in the small vicinity case via perturbation analysis and establish results that cannot be obtained by the previously studied models. For example, we prove reduction in the within-class variability of the optimized features compared to the predefined input features (via analyzing gradient flow on the "central-path" with minimal assumptions), analyze the minimizers in the near-collapse regime, and provide insights on the effect of regularization hyperparameters on the closeness to collapse. We support our theory with experiments in practical deep learning settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/16/2022

Extended Unconstrained Features Model for Exploring Deep Neural Collapse

The modern strategy for training deep neural networks for classification...
research
05/06/2021

A Geometric Analysis of Neural Collapse with Unconstrained Features

We provide the first global optimization landscape analysis of Neural Co...
research
08/05/2023

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks

Neural Collapse (NC) gives a precise description of the representations ...
research
01/21/2022

Nearest Class-Center Simplification through Intermediate Layers

Recent advances in theoretical Deep Learning have introduced geometric p...
research
09/19/2022

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

When training overparameterized deep networks for classification tasks, ...
research
08/18/2020

Prevalence of Neural Collapse during the terminal phase of deep learning training

Modern practice for training classification deepnets involves a Terminal...
research
05/22/2023

Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model

Neural collapse (NC) refers to the surprising structure of the last laye...

Please sign up or login with your details

Forgot password? Click here to reset