Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case

03/24/2021

∙

Free Probability Theory (FPT) provides rich knowledge for handling mathematical difficulties caused by random matrices that appear in research related to deep neural networks (DNNs), such as the dynamical isometry, Fisher information matrix, and training dynamics. FPT suits these researches because the DNN's parameter-Jacobian and input-Jacobian are polynomials of layerwise Jacobians. However, the critical assumption of asymptotic freenss of the layerwise Jacobian has not been proven completely so far. The asymptotic freeness assumption plays a fundamental role when propagating spectral distributions through the layers. Haar distributed orthogonal matrices are essential for achieving dynamical isometry. In this work, we prove asymptotic freeness of layerwise Jacobian of multilayer perceptrons in this case.

READ FULL TEXT

Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case

The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry

Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network

On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

Theory IIIb: Generalization in Deep Networks

Spectral properties of kernel matrices in the flat limit

Extended critical regimes of deep neural networks

Pathological spectra of the Fisher information metric and its variants in deep neural networks

Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case

Related Research

The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry

Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network

On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

Theory IIIb: Generalization in Deep Networks

Spectral properties of kernel matrices in the flat limit

Extended critical regimes of deep neural networks

Pathological spectra of the Fisher information metric and its variants in deep neural networks