Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms

by   Yazhen Wang, et al.

This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arose in statistics and machine learning where objective functions are estimated from available data. We show that these algorithms can be modeled by continuous-time ordinary or stochastic differential equations, and their asymptotic dynamic evolutions and distributions are governed by some linear ordinary or stochastic differential equations, as the data size goes to infinity. We illustrate that our study can provide a novel unified framework for a joint computational and statistical asymptotic analysis on dynamic behaviors of these algorithms with the time (or the number of iterations in the algorithms) and large sample behaviors of the statistical decision rules (like estimators and classifiers) that the algorithms are applied to compute, where the statistical decision rules are the limits of the random sequences generated from these iterative algorithms as the number of iterations goes to infinity.


page 1

page 2

page 3

page 4


Parametric estimation of stochastic differential equations via online gradient descent

We propose an online parametric estimation method of stochastic differen...

Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations

We develop the mathematical foundations of the stochastic modified equat...

Stochastic Gradient Descent for Semilinear Elliptic Equations with Uncertainties

Randomness is ubiquitous in modern engineering. The uncertainty is often...

Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent

In this paper, we study the statistical limits in terms of Sobolev norms...

Towards provably efficient quantum algorithms for large-scale machine-learning models

Large machine learning models are revolutionary technologies of artifici...

Stochastic modified equations and adaptive stochastic gradient algorithms

We develop the method of stochastic modified equations (SME), in which s...

Stochastic Langevin Differential Inclusions with Applications to Machine Learning

Stochastic differential equations of Langevin-diffusion form have receiv...

Please sign up or login with your details

Forgot password? Click here to reset