Training Language Models Using Target-Propagation

02/15/2017
by   Sam Wiseman, et al.
0

While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps. We investigate whether Target Propagation (TPROP) style approaches can address these shortcomings. Unfortunately, extensive experiments suggest that TPROP generally underperforms BPTT, and we end with an analysis of this phenomenon, and suggestions for future work.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/23/2016

Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling

Recurrent neural networks (RNNs) have shown promising performance for la...
03/16/2018

Reviving and Improving Recurrent Back-Propagation

In this paper, we revisit the recurrent back-propagation (RBP) algorithm...
04/10/2017

Bayesian Recurrent Neural Networks

In this work we explore a straightforward variational Bayes scheme for R...
10/17/2018

Online Learning of Recurrent Neural Architectures by Locally Aligning Distributed Representations

Temporal models based on recurrent neural networks have proven to be qui...
10/17/2018

Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations

Temporal models based on recurrent neural networks have proven to be qui...
06/25/2020

On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

Recurrent neural networks (RNNs) have been successfully applied to a var...
02/13/2022

Reverse Back Propagation to Make Full Use of Derivative

The development of the back-propagation algorithm represents a landmark ...

Code Repositories