DeepAI AI Chat
Log In Sign Up

Asymptotic Analysis of Deep Residual Networks

12/15/2022
by   Rama Cont, et al.
0

We investigate the asymptotic properties of deep Residual networks (ResNets) as the number of layers increases. We first show the existence of scaling regimes for trained weights markedly different from those implicitly assumed in the neural ODE literature. We study the convergence of the hidden state dynamics in these scaling regimes, showing that one may obtain an ODE, a stochastic differential equation (SDE) or neither of these. In particular, our findings point to the existence of a diffusive regime in which the deep network limit is described by a class of stochastic differential equations (SDEs). Finally, we derive the corresponding scaling limits for the backpropagation dynamics.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/25/2021

Scaling Properties of Deep Residual Networks

Residual networks (ResNets) have displayed impressive results in pattern...
06/14/2022

Scaling ResNets in the Large-depth Regime

Deep ResNets are recognized for achieving state-of-the-art results in co...
07/07/2020

Doubly infinite residual networks: a diffusion process approach

When neural network's parameters are initialized as i.i.d., neural netwo...
12/28/2021

Continuous limits of residual neural networks in case of large input data

Residual deep neural networks (ResNets) are mathematically described as ...
09/29/2021

n-Qubit Operations on Sphere and Queueing Scaling Limits for Programmable Quantum Computer

We study n-qubit operation rules on (n+1)-sphere with the target to help...
12/01/2018

Stochastic Training of Residual Networks: a Differential Equation Viewpoint

During the last few years, significant attention has been paid to the st...
05/04/2022

Asymptotic analysis of diabatic surface hopping algorithm in the adiabatic and non-adiabatic limits

Surface hopping algorithms, as an important class of quantum dynamics si...