The high-dimensional asymptotics of first order methods with random data

by   Michael Celentano, et al.

We study a class of deterministic flows in ℝ^d× k, parametrized by a random matrix X∈ℝ^n× d with i.i.d. centered subgaussian entries. We characterize the asymptotic behavior of these flows over bounded time horizons, in the high-dimensional limit in which n,d→∞ with k fixed and converging aspect ratios n/d→δ. The asymptotic characterization we prove is in terms of a system of a nonlinear stochastic process in k dimensions, whose parameters are determined by a fixed point condition. This type of characterization is known in physics as dynamical mean field theory. Rigorous results of this type have been obtained in the past for a few spin glass models. Our proof is based on time discretization and a reduction to certain iterative schemes known as approximate message passing (AMP) algorithms, as opposed to earlier work that was based on large deviations theory and stochastic processes theory. The new approach allows for a more elementary proof and implies that the high-dimensional behavior of the flow is universal with respect to the distribution of the entries of X. As specific applications, we obtain high-dimensional characterizations of gradient flow in some classical models from statistics and machine learning, under a random design assumption.



There are no comments yet.


page 1

page 2

page 3

page 4


i-flow: High-dimensional Integration and Sampling with Normalizing Flows

In many fields of science, high-dimensional integration is required. Num...

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

We analyze in a closed form the learning dynamics of stochastic gradient...

High-temperature Expansions and Message Passing Algorithms

Improved mean-field technics are a central theme of statistical physics ...

Random Features for High-Dimensional Nonlocal Mean-Field Games

We propose an efficient solution approach for high-dimensional nonlocal ...

Multi-Layer Bilinear Generalized Approximate Message Passing

In this paper, we extend the bilinear generalized approximate message pa...

Stochastic dynamical modeling of turbulent flows

Advanced measurement techniques and high performance computing have made...

Mean-field methods and algorithmic perspectives for high-dimensional machine learning

The main difficulty that arises in the analysis of most machine learning...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.