WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

04/05/2021
by   Junjie Huang, et al.
0

Producing the embedding of a sentence in an unsupervised way is valuable to natural language matching and retrieval problems in practice. In this work, we conduct a thorough examination of pretrained model based unsupervised sentence embeddings. We study on four pretrained models and conduct massive experiments on seven datasets regarding sentence semantics. We have there main findings. First, averaging all tokens is better than only using [CLS] vector. Second, combining both top andbottom layers is better than only using top layers. Lastly, an easy whitening-based vector normalization strategy with less than 10 lines of code consistently boosts the performance.

READ FULL TEXT
research
11/01/2020

Vec2Sent: Probing Sentence Embeddings with Natural Language Generation

We introspect black-box sentence embeddings by conditionally generating ...
research
08/17/2022

Neural Embeddings for Text

We propose a new kind of embedding for natural language text that deeply...
research
01/16/2019

Sentence transition matrix: An efficient approach that preserves sentence semantics

Sentence embedding is a significant research topic in the field of natur...
research
05/13/2023

A Simple and Plug-and-play Method for Unsupervised Sentence Representation Enhancement

Generating proper embedding of sentences through an unsupervised way is ...
research
05/04/2023

Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence

Sentence-level representations are beneficial for various natural langua...
research
05/10/2021

DefSent: Sentence Embeddings using Definition Sentences

Sentence embedding methods using natural language inference (NLI) datase...
research
07/31/2019

Simple Unsupervised Summarization by Contextual Matching

We propose an unsupervised method for sentence summarization using only ...

Please sign up or login with your details

Forgot password? Click here to reset