A Survey on Long Text Modeling with Transformers

02/28/2023
by   Zican Dong, et al.
0

Modeling long texts has been an essential technique in the field of natural language processing (NLP). With the ever-growing number of long documents, it is important to develop effective modeling methods that can process and analyze such texts. However, long texts pose important research challenges for existing text models, with more complex semantics and special characteristics. In this paper, we provide an overview of the recent advances on long texts modeling based on Transformer models. Firstly, we introduce the formal definition of long text modeling. Then, as the core content, we discuss how to process long input to satisfy the length limitation and design improved Transformer architectures to effectively extend the maximum context length. Following this, we discuss how to adapt Transformer models to capture the special characteristics of long texts. Finally, we describe four typical applications involving long text modeling and conclude this paper with a discussion of future directions. Our survey intends to provide researchers with a synthesis and pointer to related work on long text modeling.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/13/2020

Pretrained Transformers for Text Ranking: BERT and Beyond

The goal of text ranking is to generate an ordered list of texts retriev...
research
11/01/2020

Deep Learning for Text Attribute Transfer: A Survey

Driven by the increasingly larger deep learning models, neural language ...
research
02/22/2021

Position Information in Transformers: An Overview

Transformers are arguably the main workhorse in recent Natural Language ...
research
09/01/2022

Unsupervised Simplification of Legal Texts

The processing of legal texts has been developing as an emerging field i...
research
01/10/2022

SCROLLS: Standardized CompaRison Over Long Language Sequences

NLP benchmarks have largely focused on short texts, such as sentences an...
research
09/02/2022

Extend and Explain: Interpreting Very Long Language Models

While Transformer language models (LMs) are state-of-the-art for informa...
research
12/29/2014

Quantifying origin and character of long-range correlations in narrative texts

In natural language using short sentences is considered efficient for co...

Please sign up or login with your details

Forgot password? Click here to reset