Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings

05/18/2023
by   Qian Chen, et al.
0

Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e.g., BERT, without fine-tuning. Our analysis reveals that the sentence embeddings from BERT suffer from a bias towards uninformative words, limiting the performance in semantic textual similarity (STS) tasks. To address this bias, we propose a simple and efficient unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. Ditto can be easily applied to any pre-trained language model as a postprocessing operation. Compared to prior sentence embedding approaches, Ditto does not add parameters nor requires any learning. Empirical evaluations demonstrate that our proposed Ditto can alleviate the anisotropy problem and improve various pre-trained models on STS tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/02/2020

On the Sentence Embeddings from Pre-trained Language Models

Pre-trained contextual representations like BERT have achieved great suc...
research
04/11/2023

Towards preserving word order importance through Forced Invalidation

Large pre-trained language models such as BERT have been widely used as ...
research
05/11/2020

Toward Better Storylines with Sentence-Level Language Models

We propose a sentence-level language model which selects the next senten...
research
05/13/2023

A Simple and Plug-and-play Method for Unsupervised Sentence Representation Enhancement

Generating proper embedding of sentences through an unsupervised way is ...
research
02/07/2021

Unsupervised Sentence-embeddings by Manifold Approximation and Projection

The concept of unsupervised universal sentence encoders has gained tract...
research
02/21/2022

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

The task of abductive natural language inference (αnli), to decide which...
research
05/27/2020

Catching Attention with Automatic Pull Quote Selection

Pull quotes are an effective component of a captivating news article. Th...

Please sign up or login with your details

Forgot password? Click here to reset