DeepAI

# Towards Robust ResNet: A Small Step but A Giant Leap

This paper presents a simple yet principled approach to boosting the robustness of the residual network (ResNet) that is motivated by the dynamical system perspective. Namely, a deep neural network can be interpreted using a partial differential equation, which naturally inspires us to characterize ResNet by an explicit Euler method. Our analytical studies reveal that the step factor h in the Euler method is able to control the robustness of ResNet in both its training and generalization. Specifically, we prove that a small step factor h can benefit the training robustness for back-propagation; from the view of forward-propagation, a small h can aid in the robustness of the model generalization. A comprehensive empirical evaluation on both vision CIFAR-10 and text AG-NEWS datasets confirms that a small h aids both the training and generalization robustness.

• 22 publications
• 102 publications
• 15 publications
• 28 publications
• 57 publications
05/13/2021

### HeunNet: Extending ResNet using Heun's Methods

There is an analogy between the ResNet (Residual Network) architecture f...
04/16/2019

### On the Mathematical Understanding of ResNet with Feynman Path Integral

In this paper, we aim to understand Residual Network (ResNet) in a scien...
01/10/2021

### Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations

In this paper we consider utilizing a residual neural network (ResNet) t...
07/13/2020

### Implicit Euler ODE Networks for Single-Image Dehazing

Deep convolutional neural networks (CNN) have been applied for image deh...
03/28/2021

### Rethinking ResNets: Improved Stacking Strategies With High Order Schemes

Various Deep Neural Network architectures are keeping massive vital reco...
02/25/2018

### Functional Gradient Boosting based on Residual Network Perception

Residual Networks (ResNets) have become state-of-the-art models in deep ...
05/27/2022

### Standalone Neural ODEs with Sensitivity Analysis

This paper presents the Standalone Neural ODE (sNODE), a continuous-dept...