Do More Negative Samples Necessarily Hurt in Contrastive Learning?

05/03/2022
by   Pranjal Awasthi, et al.
0

Recent investigations in noise contrastive estimation suggest, both empirically as well as theoretically, that while having more "negative samples" in the contrastive loss improves downstream classification performance initially, beyond a threshold, it hurts downstream performance due to a "collision-coverage" trade-off. But is such a phenomenon inherent in contrastive learning? We show in a simple theoretical setting, where positive pairs are generated by sampling from the underlying latent class (introduced by Saunshi et al. (ICML 2019)), that the downstream performance of the representation optimizing the (population) contrastive loss in fact does not degrade with the number of negative samples. Along the way, we give a structural characterization of the optimal representation in our framework, for noise contrastive estimation. We also provide empirical support for our theoretical results on CIFAR-10 and CIFAR-100 datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/06/2021

Sharp Learning Bounds for Contrastive Unsupervised Representation Learning

Contrastive unsupervised representation learning (CURL) encourages data ...
research
06/18/2021

Investigating the Role of Negatives in Contrastive Representation Learning

Noise contrastive learning is a popular technique for unsupervised repre...
research
02/12/2021

Semantically-Conditioned Negative Samples for Efficient Contrastive Learning

Negative sampling is a limiting factor w.r.t. the generalization of metr...
research
03/29/2022

Contrasting the landscape of contrastive and non-contrastive learning

A lot of recent advances in unsupervised feature learning are based on d...
research
06/03/2022

Contrastive learning unifies t-SNE and UMAP

Neighbor embedding methods t-SNE and UMAP are the de facto standard for ...
research
07/06/2022

Network Binarization via Contrastive Learning

Neural network binarization accelerates deep models by quantizing their ...
research
04/04/2023

Fully Variational Noise-Contrastive Estimation

By using the underlying theory of proper scoring rules, we design a fami...

Please sign up or login with your details

Forgot password? Click here to reset