Contrastive learning, multi-view redundancy, and linear models

08/24/2020
by   Christopher Tosh, et al.
5

Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/10/2020

Demystifying Self-Supervised Learning: An Information-Theoretical Framework

Self-supervised representation learning adopts self-defined signals as s...
10/06/2021

The Power of Contrast for Feature Learning: A Theoretical Analysis

Contrastive learning has achieved state-of-the-art performance in variou...
04/27/2021

Contrastive Spatial Reasoning on Multi-View Line Drawings

Spatial reasoning on multi-view line drawings by state-of-the-art superv...
03/04/2020

Contrastive estimation reveals topic posterior information to linear models

Contrastive learning is an approach to representation learning that util...
02/17/2020

Learning Robust Representations via Multi-View Information Bottleneck

The information bottleneck principle provides an information-theoretic m...
08/15/2021

Self-supervised Contrastive Learning of Multi-view Facial Expressions

Facial expression recognition (FER) has emerged as an important componen...
08/17/2021

MVCNet: Multiview Contrastive Network for Unsupervised Representation Learning for 3D CT Lesions

Objective and Impact Statement. With the renaissance of deep learning, a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.