Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning

11/05/2017
by   Zhen He, et al.
0

Long Short-Term Memory (LSTM) is a popular approach to boosting the ability of Recurrent Neural Networks to store longer term temporal information. The capacity of an LSTM network can be increased by widening and adding layers. However, usually the former introduces additional parameters, while the latter increases the runtime. As an alternative we propose the Tensorized LSTM in which the hidden states are represented by tensors and updated via a cross-layer convolution. By increasing the tensor size, the network can be widened efficiently without additional parameters since the parameters are shared across different locations in the tensor; by delaying the output, the network can be deepened implicitly with little additional runtime since deep computations for each timestep are merged into temporal computations of the sequence. Experiments conducted on five challenging sequence learning tasks show the potential of the proposed model.

READ FULL TEXT
research
09/23/2022

Image Classification using Sequence of Pixels

This study compares sequential image classification methods based on rec...
research
12/25/2018

Tensor-Train Long Short-Term Memory for Monaural Speech Enhancement

In recent years, Long Short-Term Memory (LSTM) has become a popular choi...
research
06/22/2018

Persistent Hidden States and Nonlinear Transformation for Long Short-Term Memory

Recurrent neural networks (RNNs) have been drawing much attention with g...
research
10/19/2016

Learning to Learn Neural Networks

Meta-learning consists in learning learning algorithms. We use a Long Sh...
research
04/11/2018

Deep Differential Recurrent Neural Networks

Due to the special gating schemes of Long Short-Term Memory (LSTM), LSTM...
research
07/14/2020

Malware Detection for Forensic Memory Using Deep Recurrent Neural Networks

Memory forensics is a young but fast-growing area of research and a prom...
research
06/04/2019

Detecting Syntactic Change Using a Neural Part-of-Speech Tagger

We train a diachronic long short-term memory (LSTM) part-of-speech tagge...

Please sign up or login with your details

Forgot password? Click here to reset