Non-iterative recomputation of dense layers for performance improvement of DCNN

09/14/2018
by   Yimin Yang, et al.
10

An iterative method of learning has become a paradigm for training deep convolutional neural networks (DCNN). However, utilizing a non-iterative learning strategy can accelerate the training process of the DCNN and surprisingly such approach has been rarely explored by the deep learning (DL) community. It motivates this paper to introduce a non-iterative learning strategy that eliminates the backpropagation (BP) at the top dense or fully connected (FC) layers of DCNN, resulting in, lower training time and higher performance. The proposed method exploits the Moore-Penrose Inverse to pull back the current residual error to each FC layer, generating well-generalized features. Then using the recomputed features, i.e., the new generalized features the weights of each FC layer is computed according to the Moore-Penrose Inverse. We evaluate the proposed approach on six widely accepted object recognition benchmark datasets: Scene-15, CIFAR-10, CIFAR-100, SUN-397, Places365, and ImageNet. The experimental results show that the proposed method obtains significant improvements over 30 state-of-the-art methods. Interestingly, it also indicates that any DCNN with the proposed method can provide better performance than the same network with its original training based on BP.

READ FULL TEXT

page 4

page 5

research
06/21/2019

Fully Decoupled Neural Network Learning Using Delayed Gradients

Using the back-propagation (BP) to train neural networks requires a sequ...
research
07/31/2017

An Effective Training Method For Deep Convolutional Neural Network

In this paper, we propose the nonlinearity generation method to speed up...
research
08/13/2020

Deep Networks with Fast Retraining

Recent wor [1] has utilized Moore-Penrose (MP) inverse in deep convoluti...
research
09/23/2022

Hebbian Deep Learning Without Feedback

Recent approximations to backpropagation (BP) have mitigated many of BP'...
research
07/12/2018

Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

The backpropagation of error algorithm (BP) is often said to be impossib...
research
12/05/2020

Weight Update Skipping: Reducing Training Time for Artificial Neural Networks

Artificial Neural Networks (ANNs) are known as state-of-the-art techniqu...
research
09/20/2022

BP-Im2col: Implicit Im2col Supporting AI Backpropagation on Systolic Arrays

State-of-the-art systolic array-based accelerators adopt the traditional...

Please sign up or login with your details

Forgot password? Click here to reset