Whitening-based Contrastive Learning of Sentence Embeddings

05/28/2023
by   Wenjie Zhuo, et al.
0

This paper presents a whitening-based contrastive learning method for sentence embedding learning (WhitenedCSE), which combines contrastive learning with a novel shuffled group whitening. Generally, contrastive learning pulls distortions of a single sample (i.e., positive samples) close and push negative samples far away, correspondingly facilitating the alignment and uniformity in the feature space. A popular alternative to the "pushing” operation is whitening the feature space, which scatters all the samples for uniformity. Since the whitening and the contrastive learning have large redundancy w.r.t. the uniformity, they are usually used separately and do not easily work together. For the first time, this paper integrates whitening into the contrastive learning scheme and facilitates two benefits. 1) Better uniformity. We find that these two approaches are not totally redundant but actually have some complementarity due to different uniformity mechanism. 2) Better alignment. We randomly divide the feature into multiple groups along the channel axis and perform whitening independently within each group. By shuffling the group division, we derive multiple distortions of a single sample and thus increase the positive sample diversity. Consequently, using multiple positive samples with enhanced diversity further improves contrastive learning due to better alignment. Extensive experiments on seven semantic textual similarity tasks show our method achieves consistent improvement over the contrastive learning baseline and sets new states of the art, e.g., 78.78% (+2.53% based on BERT) Spearman correlation on STS tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/12/2023

Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

Contrastive learning-based methods, such as unsup-SimCSE, have achieved ...
research
01/16/2022

SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples

Unsupervised sentence embedding aims to obtain the most appropriate embe...
research
05/02/2022

Debiased Contrastive Learning of Unsupervised Sentence Representations

Recently, contrastive learning has been shown to be effective in improvi...
research
03/24/2023

Curricular Contrastive Regularization for Physics-aware Single Image Dehazing

Considering the ill-posed nature, contrastive regularization has been de...
research
04/07/2021

Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data

Electronic Health Record (EHR) data has been of tremendous utility in Ar...
research
10/18/2022

Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation

Contrastive self-supervised learning (CSL) with a prototypical regulariz...
research
11/10/2021

Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels

Contrastive Learning has shown impressive results on natural and medical...

Please sign up or login with your details

Forgot password? Click here to reset