Minimal Effort Back Propagation for Convolutional Neural Networks

09/18/2017
by   Bingzhen Wei, et al.
0

As traditional neural network consumes a significant amount of computing resources during back propagation, Sun2017mePropSB propose a simple yet effective technique to alleviate this problem. In this technique, only a small subset of the full gradients are computed to update the model parameters. In this paper we extend this technique into the Convolutional Neural Network(CNN) to reduce calculation in back propagation, and the surprising results verify its validity in CNN: only 5% of the gradients are passed back but the model still achieves the same effect as the traditional CNN, or even better. We also show that the top-k selection of gradients leads to a sparse calculation in back propagation, which may bring significant computational benefits for high computational complexity of convolution operation in CNN.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/17/2017

Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method

We propose a simple yet effective technique to simplify the training and...
research
06/19/2017

meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

We propose a simple yet effective technique for neural network learning....
research
10/04/2018

Learning Depth with Convolutional Spatial Propagation Network

Depth prediction is one of the fundamental problems in computer vision. ...
research
04/23/2021

GuideBP: Guiding Backpropagation Through Weaker Pathways of Parallel Logits

Convolutional neural networks often generate multiple logits and use sim...
research
07/24/2023

Entropy Transformer Networks: A Learning Approach via Tangent Bundle Data Manifold

This paper focuses on an accurate and fast interpolation approach for im...
research
09/22/2019

Performance optimization of convolution calculation by blocking and sparsity on GPU

Convolution neural network (CNN) plays a paramount role in machine learn...
research
11/07/2018

Hardware-Efficient Structure of the Accelerating Module for Implementation of Convolutional Neural Network Basic Operation

This paper presents a structural design of the hardware-efficient module...

Please sign up or login with your details

Forgot password? Click here to reset