DeepAI
Log In Sign Up

Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations

11/19/2020
by   Xinyue Huo, et al.
5

Contrastive learning has achieved great success in self-supervised visual representation learning, but existing approaches mostly ignored spatial information which is often crucial for visual representation. This paper presents heterogeneous contrastive learning (HCL), an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We demonstrate the effectiveness of HCL by showing that (i) it achieves higher accuracy in instance discrimination and (ii) it surpasses existing pre-training methods in a series of downstream tasks while shrinking the pre-training costs by half. More importantly, we show that our approach achieves higher efficiency in visual representations, and thus delivers a key message to inspire the future research of self-supervised visual representation learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/25/2021

Contrasting Contrastive Self-Supervised Representation Learning Models

In the past few years, we have witnessed remarkable breakthroughs in sel...
11/11/2022

Masked Contrastive Representation Learning

Masked image modelling (e.g., Masked AutoEncoder) and contrastive learni...
11/17/2020

Can Semantic Labels Assist Self-Supervised Visual Representation Learning?

Recently, contrastive learning has largely advanced the progress of unsu...
02/10/2022

Using Navigational Information to Learn Visual Representations

Children learn to build a visual representation of the world from unsupe...
10/30/2022

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

We introduce CAN, a simple, efficient and scalable method for self-super...
06/11/2021

Hybrid Generative-Contrastive Representation Learning

Unsupervised representation learning has recently received lots of inter...
06/17/2022

MET: Masked Encoding for Tabular Data

We consider the task of self-supervised representation learning (SSL) fo...