DeepAI
Log In Sign Up

Text Compression-aided Transformer Encoding

02/11/2021
by   Zuchao Li, et al.
6

Text encoding is one of the most important steps in Natural Language Processing (NLP). It has been done well by the self-attention mechanism in the current state-of-the-art Transformer encoder, which has brought about significant improvements in the performance of many NLP tasks. Though the Transformer encoder may effectively capture general information in its resulting representations, the backbone information, meaning the gist of the input text, is not specifically focused on. In this paper, we propose explicit and implicit text compression approaches to enhance the Transformer encoding and evaluate models using this approach on several typical downstream tasks that rely on the encoding heavily. Our explicit text compression approaches use dedicated models to compress text, while our implicit text compression approach simply adds an additional module to the main model to handle text compression. We propose three ways of integration, namely backbone source-side fusion, target-side fusion, and both-side fusion, to integrate the backbone information into Transformer-based models for various downstream tasks. Our evaluation on benchmark datasets shows that the proposed explicit and implicit text compression approaches improve results in comparison to strong baselines. We therefore conclude, when comparing the encodings to the baseline models, text compression helps the encoders to learn better language representations.

READ FULL TEXT

page 7

page 9

page 11

page 12

page 13

page 15

page 16

page 17

12/27/2019

Explicit Sentence Compression for Neural Machine Translation

State-of-the-art Transformer-based neural machine translation (NMT) syst...
10/10/2020

Information Extraction from Swedish Medical Prescriptions with Sig-Transformer Encoder

Relying on large pretrained language models such as Bidirectional Encode...
09/27/2021

Multiplicative Position-aware Transformer Models for Language Understanding

Transformer models, which leverage architectural improvements like self-...
04/30/2020

Text Segmentation by Cross Segment Attention

Document and discourse segmentation are two fundamental NLP tasks pertai...
04/15/2022

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

Many patterns in nature exhibit self-similarity: they can be compactly d...
09/26/2022

Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers

Transformer-based language models utilize the attention mechanism for su...
01/28/2023

Controlling Steering with Energy-Based Models

So-called implicit behavioral cloning with energy-based models has shown...