Adding Attentiveness to the Neurons in Recurrent Neural Networks

07/12/2018
by   Pengfei Zhang, et al.
14

Recurrent neural networks (RNNs) are capable of modeling the temporal dynamics of complex sequential information. However, the structures of existing RNN neurons mainly focus on controlling the contributions of current and historical information but do not explore the different importance levels of different elements in an input vector of a time slot. We propose adding a simple yet effective Element-wiseAttention Gate (EleAttG) to an RNN block (e.g., all RNN neurons in a network layer) that empowers the RNN neurons to have the attentiveness capability. For an RNN block, an EleAttG is added to adaptively modulate the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Specifically, the modulation of the input is content adaptive and is performed at fine granularity, being element-wise rather than input-wise. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to the action recognition tasks on both 3D human skeleton data and RGB videos. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly boosts the power of RNNs.

READ FULL TEXT

page 9

page 12

research
09/03/2019

EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks

Recurrent neural networks (RNNs) are capable of modeling temporal depend...
research
02/09/2015

Gated Feedback Recurrent Neural Networks

In this work, we propose a novel recurrent neural network (RNN) architec...
research
03/13/2018

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Recurrent neural networks (RNNs) have been widely used for processing se...
research
10/22/2018

Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Recurrent neural network (RNN) models are widely used for processing seq...
research
06/10/2017

Image Matching via Loopy RNN

Most existing matching algorithms are one-off algorithms, i.e., they usu...
research
11/24/2015

rnn : Recurrent Library for Torch

The rnn package provides components for implementing a wide range of Rec...
research
09/13/2021

Applications of Recurrent Neural Network for Biometric Authentication Anomaly Detection

Recurrent Neural Networks are powerful machine learning frameworks that ...

Please sign up or login with your details

Forgot password? Click here to reset