Unsupervised Document Embedding via Contrastive Augmentation

by   Dongsheng Luo, et al.

We present a contrasting learning approach with data augmentation techniques to learn document representations in an unsupervised manner. Inspired by recent contrastive self-supervised learning algorithms used for image and NLP pretraining, we hypothesize that high-quality document embedding should be invariant to diverse paraphrases that preserve the semantics of the original document. With different backbones and contrastive learning frameworks, our study reveals the enormous benefits of contrastive augmentation for document representation learning with two additional insights: 1) including data augmentation in a contrastive way can substantially improve the embedding quality in unsupervised document representation learning, and 2) in general, stochastic augmentations generated by simple word-level manipulation work much better than sentence-level and document-level ones. We plug our method into a classifier and compare it with a broad range of baseline methods on six benchmark datasets. Our method can decrease the classification error rate by up to 6.4 or even surpassing fully-supervised methods.


Collaborative Graph Contrastive Learning: Data Augmentation Composition May Not be Necessary for Graph Representation Learning

Unsupervised graph representation learning is a non-trivial topic for gr...

Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning

We introduce EfficientCL, a memory-efficient continual pretraining metho...

Spiral Contrastive Learning: An Efficient 3D Representation Learning Method for Unannotated CT Lesions

Computed tomography (CT) samples with pathological annotations are diffi...

DU-Net based Unsupervised Contrastive Learning for Cancer Segmentation in Histology Images

In this paper, we introduce an unsupervised cancer segmentation framewor...

Center-wise Local Image Mixture For Contrastive Representation Learning

Recent advances in unsupervised representation learning have experienced...

Watching the World Go By: Representation Learning from Unlabeled Videos

Recent single image unsupervised representation learning techniques show...

Augmentations in Graph Contrastive Learning: Current Methodological Flaws Towards Better Practices

Graph classification has applications in bioinformatics, social sciences...