Forward Stability of ResNet and Its Variants

11/24/2018
by   Linan Zhang, et al.
0

The residual neural network (ResNet) is a popular deep network architecture which has the ability to obtain high-accuracy results on several image processing problems. In order to analyze the behavior and structure of ResNet, recent work has been on establishing connections between ResNets and continuous-time optimal control problems. In this work, we show that the post-activation ResNet is related to an optimal control problem with differential inclusions, and provide continuous-time stability results for the differential inclusion associated with ResNet. Motivated by the stability conditions, we show that alterations of either the architecture or the optimization problem can generate variants of ResNet which improve the theoretical stability bounds. In addition, we establish stability bounds for the full (discrete) network associated with two variants of ResNet, in particular, bounds on the growth of the features and a measure of the sensitivity of the features with respect to perturbations. These results also help to show the relationship between the depth, regularization, and stability of the feature space. Computational experiments on the proposed variants show that the accuracy of ResNet is preserved and that the accuracy seems to be monotone with respect to the depth and various corruptions.

READ FULL TEXT

page 14

page 21

page 22

research
05/13/2021

HeunNet: Extending ResNet using Heun's Methods

There is an analogy between the ResNet (Residual Network) architecture f...
research
12/18/2019

ResNetX: a more disordered and deeper network architecture

Designing efficient network structures has always been the core content ...
research
06/22/2022

Near-optimal control of dynamical systems with neural ordinary differential equations

Optimal control problems naturally arise in many scientific applications...
research
05/07/2020

An Optimal Control Theory for the Traveling Salesman Problem and Its Variants

We show that the traveling salesman problem (TSP) and its many variants ...
research
01/31/2022

Imbedding Deep Neural Networks

Continuous depth neural networks, such as Neural ODEs, have refashioned ...
research
04/16/2019

On the Mathematical Understanding of ResNet with Feynman Path Integral

In this paper, we aim to understand Residual Network (ResNet) in a scien...
research
06/25/2023

A Neural RDE approach for continuous-time non-Markovian stochastic control problems

We propose a novel framework for solving continuous-time non-Markovian s...

Please sign up or login with your details

Forgot password? Click here to reset