Log In Sign Up

Contrasting Contrastive Self-Supervised Representation Learning Models

by   Klemen Kotar, et al.

In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning. Despite the success and adoption of representations learned through this paradigm, much is yet to be understood about how different training methods and datasets influence performance on downstream tasks. In this paper, we analyze contrastive approaches as one of the most successful and popular variants of self-supervised representation learning. We perform this analysis from the perspective of the training algorithms, pre-training datasets and end tasks. We examine over 700 training experiments including 30 encoders, 4 pre-training datasets and 20 diverse downstream tasks. Our experiments address various questions regarding the performance of self-supervised models compared to their supervised counterparts, current benchmarks used for evaluation, and the effect of the pre-training data on end task performance. We hope the insights and empirical evidence provided by this work will help future research in learning better visual representations.


page 2

page 3

page 9

page 10

page 12

page 13

page 14

page 16


Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations

Contrastive learning has achieved great success in self-supervised visua...

Similarity Analysis of Self-Supervised Speech Representations

Self-supervised speech representation learning has recently been a prosp...

A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Tremendous progress has been made in visual representation learning, not...

On the robustness of self-supervised representations for multi-view object classification

It is known that representations from self-supervised pre-training can p...

Self-Supervised Visual Representation Learning Using Lightweight Architectures

In self-supervised learning, a model is trained to solve a pretext task,...

Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

Instance discriminative self-supervised representation learning has been...

Code Repositories


Visual Representation Learning Benchmark for Self-Supervised Models

view repo