DeepAI AI Chat
Log In Sign Up

Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra

by   Vardan Papyan, et al.

Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers. We identify and discuss an important formal class/cross-class structure and show how it lies at the origin of the many visually striking features observed in deepnet spectra, some of which were reported in recent articles, others are unveiled here for the first time. These include spectral outliers, "spikes", and small but distinct continuous distributions, "bumps", often seen beyond the edge of a "main bulk". The significance of the cross-class structure is illustrated in three ways: (i) we prove the ratio of outliers to bulk in the spectrum of the Fisher information matrix is predictive of misclassification, in the context of multinomial logistic regression; (ii) we demonstrate how, gradually with depth, a network is able to separate class-distinctive information from class variability, all while orthogonalizing the class-distinctive information; and (iii) we propose a correction to KFAC, a well-known second-order optimization algorithm for training deepnets.


Beyond Random Matrix Theory for Deep Networks

We investigate whether the Wigner semi-circle and Marcenko-Pastur distri...

On the Variance of the Fisher Information for Deep Learning

The Fisher information matrix (FIM) has been applied to the realm of dee...

Cross-Spectral Periocular Recognition with Conditional Adversarial Networks

This work addresses the challenge of comparing periocular images capture...

Removing grid structure in angle-resolved photoemission spectra via deep learning method

Spectroscopic data may often contain unwanted extrinsic signals. For exa...

Active deep learning method for the discovery of objects of interest in large spectroscopic surveys

Current archives of the LAMOST telescope contain millions of pipeline-pr...

Spectrum of non-Hermitian deep-Hebbian neural networks

Neural networks with recurrent asymmetric couplings are important to und...

Decoding Structure-Spectrum Relationships with Physically Organized Latent Spaces

A new semi-supervised machine learning method for the discovery of struc...