DeepAI AI Chat
Log In Sign Up

Random matrix analysis of deep neural network weight matrices

by   Matthias Thamm, et al.

Neural networks have been used successfully in a variety of fields, which has led to a great deal of interest in developing a theoretical understanding of how they store the information needed to perform a particular task. We study the weight matrices of trained deep neural networks using methods from random matrix theory (RMT) and show that the statistics of most of the singular values follow universal RMT predictions. This suggests that they are random and do not contain system specific information, which we investigate further by comparing the statistics of eigenvector entries to the universal Porter-Thomas distribution. We find that for most eigenvectors the hypothesis of randomness cannot be rejected, and that only eigenvectors belonging to the largest singular values deviate from the RMT prediction, indicating that they may encode learned information. We analyze the spectral distribution of such large singular values using the Hill estimator and find that the distribution cannot be characterized by a tail index, i.e. is not of power law type.


page 1

page 2

page 3

page 4


Boundary between noise and information applied to filtering neural network weight matrices

Deep neural networks have been successfully applied to a broad range of ...

On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

We study the distribution of singular values of product of random matric...

The asymptotic joint distribution of the largest and the smallest singular values for random circulant matrices

In this manuscript, we study the limiting distribution for the joint law...

Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case

The paper deals with the distribution of singular values of the input-ou...

Hashing-like Johnson–Lindenstrauss transforms and their extreme singular values

The Johnson–Lindenstrauss (JL) lemma is a powerful tool for dimensionali...

Universal characteristics of deep neural network loss surfaces from random matrix theory

This paper considers several aspects of random matrix universality in de...

Koopman-Based Bound for Generalization: New Aspect of Neural Networks Regarding Nonlinear Noise Filtering

We propose a new bound for generalization of neural networks using Koopm...