Identity Mappings in Deep Residual Networks

03/16/2016
by   Kaiming He, et al.
1

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62 Code is available at: https://github.com/KaimingHe/resnet-1k-layers

READ FULL TEXT

page 1

page 2

page 3

page 4

11/04/2016

Learning Identity Mappings with Residual Gates

We propose a new layer design by adding a linear gating mechanism to sho...
05/18/2018

Norm-Preservation: Why Residual Networks Can Become Extremely Deep?

Augmenting deep neural networks with skip connections, as introduced in ...
11/01/2021

RMNet: Equivalently Removing Residual Connection from Networks

Although residual connection enables training very deep neural networks,...
09/29/2017

Deep Competitive Pathway Networks

In the design of deep neural architectures, recent studies have demonstr...
11/23/2016

Deep Convolutional Neural Networks with Merge-and-Run Mappings

A deep residual network, built by stacking a sequence of residual blocks...
05/24/2019

Robust learning with implicit residual networks

In this effort we propose a new deep architecture utilizing residual blo...

Code Repositories

resnet-1k-layers

Deep Residual Networks with 1K Layers


view repo

lasagne_residual_network

Recreating the Deep Residual Network in Lasagne


view repo

resnet-cifar10

ResNet for Cifar10


view repo

matconvnet-ResNet

MatConvNet Implementation for Deep Residual Networks


view repo

matconvnet-ResNet

Deep Residual Networks for MatConvNet


view repo

Please sign up or login with your details

Forgot password? Click here to reset