DeepAI AI Chat
Log In Sign Up

Towards Robust ResNet: A Small Step but A Giant Leap

by   Jingfeng Zhang, et al.

This paper presents a simple yet principled approach to boosting the robustness of the residual network (ResNet) that is motivated by the dynamical system perspective. Namely, a deep neural network can be interpreted using a partial differential equation, which naturally inspires us to characterize ResNet by an explicit Euler method. Our analytical studies reveal that the step factor h in the Euler method is able to control the robustness of ResNet in both its training and generalization. Specifically, we prove that a small step factor h can benefit the training robustness for back-propagation; from the view of forward-propagation, a small h can aid in the robustness of the model generalization. A comprehensive empirical evaluation on both vision CIFAR-10 and text AG-NEWS datasets confirms that a small h aids both the training and generalization robustness.


page 9

page 17


HeunNet: Extending ResNet using Heun's Methods

There is an analogy between the ResNet (Residual Network) architecture f...

On the Mathematical Understanding of ResNet with Feynman Path Integral

In this paper, we aim to understand Residual Network (ResNet) in a scien...

Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations

In this paper we consider utilizing a residual neural network (ResNet) t...

Implicit Euler ODE Networks for Single-Image Dehazing

Deep convolutional neural networks (CNN) have been applied for image deh...

Rethinking ResNets: Improved Stacking Strategies With High Order Schemes

Various Deep Neural Network architectures are keeping massive vital reco...

Functional Gradient Boosting based on Residual Network Perception

Residual Networks (ResNets) have become state-of-the-art models in deep ...

Standalone Neural ODEs with Sensitivity Analysis

This paper presents the Standalone Neural ODE (sNODE), a continuous-dept...