A Survey on Model Compression for Natural Language Processing

02/15/2022
by   Canwen Xu, et al.
0

With recent developments in new architectures like Transformer and pretraining techniques, significant progress has been made in applications of natural language processing (NLP). However, the high energy cost and long inference delay of Transformer is preventing NLP from entering broader scenarios including edge and mobile computing. Efficient NLP research aims to comprehensively consider computation, time and carbon emission for the entire life-cycle of NLP, including data preparation, model training and inference. In this survey, we focus on the inference stage and review the current state of model compression for NLP, including the benchmarks, metrics and methodology. We outline the current obstacles and future research directions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/15/2022

A Survey on Dynamic Neural Networks for Natural Language Processing

Effectively scaling large Transformer models is a main driver of recent ...
research
07/19/2023

Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation

Rising computational demands of modern natural language processing (NLP)...
research
04/30/2021

Summarization, Simplification, and Generation: The Case of Patents

We survey Natural Language Processing (NLP) approaches to summarizing, s...
research
04/19/2020

The Cost of Training NLP Models: A Concise Overview

We review the cost of training large-scale language models, and the driv...
research
07/08/2021

A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models

Bangla – ranked as the 6th most widely spoken language across the world ...
research
11/08/2021

A Survey on Green Deep Learning

In recent years, larger and deeper models are springing up and continuou...
research
08/27/2023

Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models

Generative pre-trained transformer (GPT) models have revolutionized the ...

Please sign up or login with your details

Forgot password? Click here to reset