Reviving and Improving Recurrent Back-Propagation

03/16/2018
by   Renjie Liao, et al.
0

In this paper, we revisit the recurrent back-propagation (RBP) algorithm, discuss the conditions under which it applies as well as how to satisfy them in deep neural networks. We show that RBP can be unstable and propose two variants based on conjugate gradient on the normal equations (CG-RBP) and Neumann series (Neumann-RBP). We further investigate the relationship between Neumann-RBP and back propagation through time (BPTT) and its truncated version (TBPTT). Our Neumann-RBP has the same time complexity as TBPTT but only requires constant memory, whereas TBPTT's memory cost scales linearly with the number of truncation steps. We examine all RBP variants along with BPTT and TBPTT in three different application domains: associative memory with continuous Hopfield networks, document classification in citation networks using graph neural networks and hyperparameter optimization for fully connected networks. All experiments demonstrate that RBPs, especially the Neumann-RBP variant, are efficient and effective for optimizing convergent recurrent neural networks.

READ FULL TEXT

page 6

page 7

page 13

research
02/15/2017

Training Language Models Using Target-Propagation

While Truncated Back-Propagation through Time (BPTT) is the most popular...
research
04/06/2020

Verifying Recurrent Neural Networks using Invariant Inference

Deep neural networks are revolutionizing the way complex systems are dev...
research
03/03/2021

Recurrent Graph Neural Network Algorithm for Unsupervised Network Community Detection

Network community detection often relies on optimizing partition quality...
research
02/06/2019

Compression of Recurrent Neural Networks for Efficient Language Modeling

Recurrent neural networks have proved to be an effective method for stat...
research
12/07/2021

CCasGNN: Collaborative Cascade Prediction Based on Graph Neural Networks

Cascade prediction aims at modeling information diffusion in the network...
research
10/24/2022

OLLA: Decreasing the Memory Usage of Neural Networks by Optimizing the Lifetime and Location of Arrays

The size of deep neural networks has grown exponentially in recent years...
research
05/22/2020

Stable and expressive recurrent vision models

Primate vision depends on recurrent processing for reliable perception (...

Please sign up or login with your details

Forgot password? Click here to reset