The Reversible Residual Network: Backpropagation Without Storing Activations

07/14/2017
by   Aidan N. Gomez, et al.
0

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/14/2019

A Partially Reversible U-Net for Memory-Efficient Volumetric Image Segmentation

One of the key drawbacks of 3D convolutional neural networks for segment...
research
09/12/2017

Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Recently, deep residual networks have been successfully applied in many ...
research
01/27/2023

Deep Residual Compensation Convolutional Network without Backpropagation

PCANet and its variants provided good accuracy results for classificatio...
research
06/15/2023

PaReprop: Fast Parallelized Reversible Backpropagation

The growing size of datasets and deep learning models has made faster an...
research
04/19/2021

Memory Efficient 3D U-Net with Reversible Mobile Inverted Bottlenecks for Brain Tumor Segmentation

We propose combining memory saving techniques with traditional U-Net arc...
research
01/26/2022

Momentum Capsule Networks

Capsule networks are a class of neural networks that achieved promising ...
research
06/28/2022

RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network

This work introduces the RevSilo, the first reversible module for bidire...

Please sign up or login with your details

Forgot password? Click here to reset