Structured Pruning of Recurrent Neural Networks through Neuron Selection

06/17/2019
by   Liangjiang Wen, et al.
5

Recurrent neural networks (RNNs) have recently achieved remarkable successes in a number of applications. However, the huge sizes and computational burden of these models make it difficult for their deployment on edge devices. A practically effective approach is to reduce the overall storage and computation costs of RNNs by network pruning techniques. Despite their successful applications, those pruning methods based on Lasso either produce irregular sparse patterns in weight matrices, which is not helpful in practical speedup. To address these issues, we propose structured pruning method through neuron selection which can reduce the sizes of basic structures of RNNs. More specifically, we introduce two sets of binary random variables, which can be interpreted as gates or switches to the input neurons and the hidden neurons, respectively. We demonstrate that the corresponding optimization problem can be addressed by minimizing the L0 norm of the weight matrix. Finally, experimental results on language modeling and machine reading comprehension tasks have indicated the advantages of the proposed method in comparison with state-of-the-art pruning competitors. In particular, nearly 20 x practical speedup during inference was achieved without losing performance for language model on the Penn TreeBank dataset, indicating the promising performance of the proposed method

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/23/2021

Spectral Pruning for Recurrent Neural Networks

Pruning techniques for neural networks with a recurrent architecture, su...
research
03/20/2018

Efficient Recurrent Neural Networks using Structured Matrices in FPGAs

Recurrent Neural Networks (RNNs) are becoming increasingly important for...
research
08/20/2017

Neural Networks Compression for Language Modeling

In this paper, we consider several compression techniques for the langua...
research
11/30/2019

One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation

Recent advances in the sparse neural network literature have made it pos...
research
11/13/2019

Structured Sparsification of Gated Recurrent Neural Networks

Recently, a lot of techniques were developed to sparsify the weights of ...
research
01/14/2020

Block-wise Dynamic Sparseness

Neural networks have achieved state of the art performance across a wide...
research
10/01/2022

EAPruning: Evolutionary Pruning for Vision Transformers and CNNs

Structured pruning greatly eases the deployment of large neural networks...

Please sign up or login with your details

Forgot password? Click here to reset