Contrastive Learning with Hard Negative Samples

10/09/2020
by   Joshua Robinson, et al.
0

We consider the question: how can you sample good negative examples for contrastive learning? We argue that, as with metric learning, learning contrastive representations benefits from hard negative samples (i.e., points that are difficult to distinguish from an anchor point). The key challenge toward using hard negatives is that contrastive methods must remain unsupervised, making it infeasible to adopt existing negative sampling strategies that use label information. In response, we develop a new class of unsupervised methods for selecting hard negative samples where the user can control the amount of hardness. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.

READ FULL TEXT
research
06/02/2022

Hard Negative Sampling Strategies for Contrastive Representation Learning

One of the challenges in contrastive learning is the selection of approp...
research
04/22/2022

Universum-inspired Supervised Contrastive Learning

Mixup is an efficient data augmentation method which generates additiona...
research
11/04/2021

Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning

We study the problem of designing hard negative sampling distributions f...
research
02/15/2018

Learning Determinantal Point Processes by Sampling Inferred Negatives

Determinantal Point Processes (DPPs) have attracted significant interest...
research
02/12/2021

Semantically-Conditioned Negative Samples for Efficient Contrastive Learning

Negative sampling is a limiting factor w.r.t. the generalization of metr...
research
10/23/2022

Tail Batch Sampling: Approximating Global Contrastive Losses as Optimization over Batch Assignments

Contrastive Learning has recently achieved state-of-the-art performance ...
research
03/21/2023

Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation

Cross-View Geo-Localisation is still a challenging task where additional...

Please sign up or login with your details

Forgot password? Click here to reset