Log In Sign Up

Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations

by   Xinyue Huo, et al.

Contrastive learning has achieved great success in self-supervised visual representation learning, but existing approaches mostly ignored spatial information which is often crucial for visual representation. This paper presents heterogeneous contrastive learning (HCL), an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We demonstrate the effectiveness of HCL by showing that (i) it achieves higher accuracy in instance discrimination and (ii) it surpasses existing pre-training methods in a series of downstream tasks while shrinking the pre-training costs by half. More importantly, we show that our approach achieves higher efficiency in visual representations, and thus delivers a key message to inspire the future research of self-supervised visual representation learning.


page 1

page 2

page 3

page 4


Contrasting Contrastive Self-Supervised Representation Learning Models

In the past few years, we have witnessed remarkable breakthroughs in sel...

Masked Contrastive Representation Learning

Masked image modelling (e.g., Masked AutoEncoder) and contrastive learni...

Can Semantic Labels Assist Self-Supervised Visual Representation Learning?

Recently, contrastive learning has largely advanced the progress of unsu...

Using Navigational Information to Learn Visual Representations

Children learn to build a visual representation of the world from unsupe...

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

We introduce CAN, a simple, efficient and scalable method for self-super...

Hybrid Generative-Contrastive Representation Learning

Unsupervised representation learning has recently received lots of inter...

MET: Masked Encoding for Tabular Data

We consider the task of self-supervised representation learning (SSL) fo...