Gated Feedback Recurrent Neural Networks

by   Junyoung Chung, et al.

In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.


Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

In this paper we compare different types of recurrent units in recurrent...

Gated Recurrent Neural Tensor Network

Recurrent Neural Networks (RNNs), which are a powerful scheme for modeli...

How to Construct Deep Recurrent Neural Networks

In this paper, we explore different ways to extend a recurrent neural ne...

A New Hybrid-parameter Recurrent Neural Networks for Online Handwritten Chinese Character Recognition

The recurrent neural network (RNN) is appropriate for dealing with tempo...

A single-layer RNN can approximate stacked and bidirectional RNNs, and topologies in between

To enhance the expressiveness and representational capacity of recurrent...

Direct detection of pixel-level myocardial infarction areas via a deep-learning algorithm

Accurate detection of the myocardial infarction (MI) area is crucial for...

Adding Attentiveness to the Neurons in Recurrent Neural Networks

Recurrent neural networks (RNNs) are capable of modeling the temporal dy...

Code Repositories


A Practical Implementation of Gated Feedback Recurrent Neural Network for both LSTM and GRU using Tensorflow:

view repo