DeepAI AI Chat
Log In Sign Up

Robust learning with implicit residual networks

by   Viktor Reshniak, et al.
Oak Ridge National Laboratory
The University of Tennessee, Knoxville

In this effort we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to improved stability of both forward and backward propagations, has a favorable impact on the generalization power of the network and allows for higher learning rates. In addition, we consider a reformulation of ResNet which does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability and robustness. Finally, we derive the memory efficient reversible training algorithm and provide numerical results in support of our findings.


page 7

page 8


m-RevNet: Deep Reversible Neural Networks with Momentum

In recent years, the connections between deep residual networks and firs...

Identity Mappings in Deep Residual Networks

Deep residual networks have emerged as a family of extremely deep archit...

Doubly infinite residual networks: a diffusion process approach

When neural network's parameters are initialized as i.i.d., neural netwo...

CrescendoNet: A Simple Deep Convolutional Neural Network with Ensemble Behavior

We introduce a new deep convolutional neural network, CrescendoNet, by s...

Meta-Aggregating Networks for Class-Incremental Learning

Class-Incremental Learning (CIL) aims to learn a classification model wi...

Invertible Residual Networks

Reversible deep networks provide useful theoretical guarantees and have ...

ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees

Models recently used in the literature proving residual networks (ResNet...