Compression of Deep Learning Models for Text: A Survey

by   Manish Gupta, et al.

In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanks to deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs) networks, and Transformer based models like Bidirectional Encoder Representations from Transformers (BERT). But these models are humongous in size. On the other hand, real world applications demand small model size, low response times and low computational power wattage. In this survey, we discuss six different types of methods (Pruning, Quantization, Knowledge Distillation, Parameter Sharing, Tensor Decomposition, and Linear Transformer based methods) for compression of such models to enable their deployment in real industry NLP projects. Given the critical need of building applications with efficient and small models, and the large amount of recently published work in this area, we believe that this survey organizes the plethora of work done by the 'deep learning for NLP' community in the past few years and presents it as a coherent story.


page 1

page 2

page 3

page 4


Knowledge Distillation: A Survey

In recent years, deep neural networks have been very successful in the f...

A Survey on Green Deep Learning

In recent years, larger and deeper models are springing up and continuou...

The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

Deep learning has demonstrated tremendous success in variety of applicat...

A Practical Survey on Faster and Lighter Transformers

Recurrent neural networks are effective models to process sequences. How...

An Overview of Neural Network Compression

Overparameterized networks trained to convergence have shown impressive ...

A Survey of Model Compression and Acceleration for Deep Neural Networks

Deep convolutional neural networks (CNNs) have recently achieved great s...

Classification Models for Partially Ordered Sequences

Many models such as Long Short Term Memory (LSTMs), Gated Recurrent Unit...