Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations

04/06/2022
by   Jeff Z. HaoChen, et al.
0

Contrastive learning is a highly effective method which uses unlabeled data to produce representations which are linearly separable for downstream classification tasks. Recent works have shown that contrastive representations are not only useful when data come from a single domain, but are also effective for transferring across domains. Concretely, when contrastive representations are trained on data from two domains (a source and target) and a linear classification head is trained to predict labels using only the labeled source data, the resulting classifier also exhibits good transfer to the target domain. In this work, we analyze this linear transferability phenomenon, building upon the framework proposed by HaoChen et al (2021) which relates contrastive learning to spectral clustering of a positive-pair graph on the data. We prove that contrastive representations capture relationships between subpopulations in the positive-pair graph: linear transferability can occur when data from the same class in different domains (e.g., photo dogs and cartoon dogs) are connected in the graph. Our analysis allows the source and target classes to have unbounded density ratios and be mapped to distant representations. Our proof is also built upon technical improvements over the main results of HaoChen et al (2021), which may be of independent interest.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/24/2021

A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Tremendous progress has been made in visual representation learning, not...
research
11/24/2022

Cross-domain Transfer of defect features in technical domains based on partial target data

A common challenge in real world classification scenarios with sequentia...
research
12/23/2019

How to Pick the Best Source Data? Measuring Transferability for Heterogeneous Domains

Given a set of source data with pre-trained classification models, how c...
research
07/31/2020

Adversarial Bipartite Graph Learning for Video Domain Adaptation

Domain adaptation techniques, which focus on adapting models between dis...
research
03/22/2016

Enhanced perceptrons using contrastive biclusters

Perceptrons are neuronal devices capable of fully discriminating linearl...
research
02/06/2022

Low-confidence Samples Matter for Domain Adaptation

Domain adaptation (DA) aims to transfer knowledge from a label-rich sour...
research
08/21/2019

Transferability and Hardness of Supervised Classification Tasks

We propose a novel approach for estimating the difficulty and transferab...

Please sign up or login with your details

Forgot password? Click here to reset