DeepAI AI Chat
Log In Sign Up

Asymptotic Analysis of Deep Residual Networks

by   Rama Cont, et al.

We investigate the asymptotic properties of deep Residual networks (ResNets) as the number of layers increases. We first show the existence of scaling regimes for trained weights markedly different from those implicitly assumed in the neural ODE literature. We study the convergence of the hidden state dynamics in these scaling regimes, showing that one may obtain an ODE, a stochastic differential equation (SDE) or neither of these. In particular, our findings point to the existence of a diffusive regime in which the deep network limit is described by a class of stochastic differential equations (SDEs). Finally, we derive the corresponding scaling limits for the backpropagation dynamics.


page 1

page 2

page 3

page 4


Scaling Properties of Deep Residual Networks

Residual networks (ResNets) have displayed impressive results in pattern...

Scaling ResNets in the Large-depth Regime

Deep ResNets are recognized for achieving state-of-the-art results in co...

Doubly infinite residual networks: a diffusion process approach

When neural network's parameters are initialized as i.i.d., neural netwo...

Continuous limits of residual neural networks in case of large input data

Residual deep neural networks (ResNets) are mathematically described as ...

n-Qubit Operations on Sphere and Queueing Scaling Limits for Programmable Quantum Computer

We study n-qubit operation rules on (n+1)-sphere with the target to help...

Stochastic Training of Residual Networks: a Differential Equation Viewpoint

During the last few years, significant attention has been paid to the st...

Asymptotic analysis of diabatic surface hopping algorithm in the adiabatic and non-adiabatic limits

Surface hopping algorithms, as an important class of quantum dynamics si...