Learning Compact Recurrent Neural Networks

04/09/2016
by   Zhiyun Lu, et al.
0

Recurrent neural networks (RNNs), including long short-term memory (LSTM) RNNs, have produced state-of-the-art results on a variety of speech recognition tasks. However, these models are often too large in size for deployment on mobile devices with memory and latency constraints. In this work, we study mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and parameter sharing schemes. Our goal is to investigate redundancies in recurrent architectures where compression can be admitted without losing performance. A hybrid strategy of using structured matrices in the bottom layers and shared low-rank factors on the top layers is found to be particularly effective, reducing the parameters of a standard LSTM by 75 increase in WER, on a 2,000-hr English Voice Search task.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/25/2016

On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition

We study the problem of compressing recurrent neural networks (RNNs). In...
research
01/07/2018

Approximate FPGA-based LSTMs under Computation Time Constraints

Recurrent Neural Networks and in particular Long Short-Term Memory (LSTM...
research
05/03/2022

Biometric Signature Verification Using Recurrent Neural Networks

Architectures based on Recurrent Neural Networks (RNNs) have been succes...
research
08/24/2017

Learning the Enigma with Recurrent Neural Networks

Recurrent neural networks (RNNs) represent the state of the art in trans...
research
12/02/2016

Parameter Compression of Recurrent Neural Networks and Degradation of Short-term Memory

The significant computational costs of deploying neural networks in larg...
research
09/15/2017

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Model compression is significant for the wide adoption of Recurrent Neur...
research
04/26/2018

Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip

Recurrent Neural Networks (RNNs) are powerful tools for solving sequence...

Please sign up or login with your details

Forgot password? Click here to reset