Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks

by   Lucas Benigni, et al.

This paper is concerned with the asymptotic distribution of the largest eigenvalues for some nonlinear random matrix ensemble stemming from the study of neural networks. More precisely we consider M= 1/m YY^⊤ with Y=f(WX) where W and X are random rectangular matrices with i.i.d. centered entries. This models the data covariance matrix or the Conjugate Kernel of a single layered random Feed-Forward Neural Network. The function f is applied entrywise and can be seen as the activation function of the neural network. We show that the largest eigenvalue has the same limit (in probability) as that of some well-known linear random matrix ensembles. In particular, we relate the asymptotic limit of the largest eigenvalue for the nonlinear model to that of an information-plus-noise random matrix, establishing a possible phase transition depending on the function f and the distribution of W and X. This may be of interest for applications to machine learning.


Eigenvalue distribution of nonlinear models of random matrices

This paper is concerned with the asymptotic empirical eigenvalue distrib...

Large deviations for the largest eigenvalues and eigenvectors of spiked random matrices

We consider matrices formed by a random N× N matrix drawn from the Gauss...

Eigenvector distribution in the critical regime of BBP transition

In this paper, we study the random matrix model of Gaussian Unitary Ense...

Convergence rate to the Tracy–Widom laws for the largest eigenvalue of sample covariance matrices

We establish a quantitative version of the Tracy–Widom law for the large...

Phase Transition in the Generalized Stochastic Block Model

We study the problem of detecting the community structure from the gener...

Computable structural formulas for the distribution of the β-Jacobi edge eigenvalues

The Jacobi ensemble is one of the classical ensembles of random matrix t...

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

The logit outputs of a feedforward neural network at initialization are ...