DeepAI AI Chat
Log In Sign Up

Universality Laws for High-Dimensional Learning with Random Features

by   Hong Hu, et al.

We prove a universality theorem for learning with random features. Our result shows that, in terms of training and generalization errors, the random feature model with a nonlinear activation function is asymptotically equivalent to a surrogate Gaussian model with a matching covariance matrix. This settles a conjecture based on which several recent papers develop their results. Our method for proving the universality builds on the classical Lindeberg approach. Major ingredients of the proof include a leave-one-out analysis for the optimization problem associated with the training process and a central limit theorem, obtained via Stein's method, for weakly correlated random variables.


page 1

page 2

page 3

page 4


What Intraclass Covariance Structures Can Symmetric Bernoulli Random Variables Have?

The covariance matrix of random variables X_1,…,X_n is said to have an i...

Central Limit Theorem for Linear Eigenvalue Statistics for Submatrices of Wigner Random Matrices

We prove the Central Limit Theorem for finite-dimensional vectors of lin...

Convergence rate to the Tracy–Widom laws for the largest eigenvalue of sample covariance matrices

We establish a quantitative version of the Tracy–Widom law for the large...

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

The logit outputs of a feedforward neural network at initialization are ...

A Central Limit Theorem for incomplete U-statistics over triangular arrays

We analyze the fluctuations of incomplete U-statistics over a triangular...

On the Inherent Regularization Effects of Noise Injection During Training

Randomly perturbing networks during the training process is a commonly u...

Deterministic equivalent and error universality of deep random features learning

This manuscript considers the problem of learning a random Gaussian netw...