Semantically-Conditioned Negative Samples for Efficient Contrastive Learning

02/12/2021
by   James O'Neill, et al.
26

Negative sampling is a limiting factor w.r.t. the generalization of metric-learned neural networks. We show that uniform negative sampling provides little information about the class boundaries and thus propose three novel techniques for efficient negative sampling: drawing negative samples from (1) the top-k most semantically similar classes, (2) the top-k most semantically similar samples and (3) interpolating between contrastive latent representations to create pseudo negatives. Our experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet-200 show that our proposed Semantically Conditioned Negative Sampling and Latent Mixup lead to consistent performance improvements. In the standard supervised learning setting, on average we increase test accuracy by 1.52% percentage points on CIFAR-10 across various network architectures. In the knowledge distillation setting, (1) the performance of student networks increase by 4.56% percentage points on Tiny-ImageNet-200 and 3.29% on CIFAR-100 over student networks trained with no teacher and (2) 1.23% and 1.72% respectively over a hard-to-beat baseline (Hinton et al., 2015).

READ FULL TEXT

page 7

page 17

page 18

page 19

page 20

05/03/2022

Do More Negative Samples Necessarily Hurt in Contrastive Learning?

Recent investigations in noise contrastive estimation suggest, both empi...
02/25/2019

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Recent empirical works have successfully used unlabeled data to learn fe...
10/09/2020

Contrastive Learning with Hard Negative Samples

We consider the question: how can you sample good negative examples for ...
03/18/2021

Similarity Transfer for Knowledge Distillation

Knowledge distillation is a popular paradigm for learning portable neura...
07/01/2020

Debiased Contrastive Learning

A prominent technique for self-supervised representation learning has be...
07/16/2022

Model-Aware Contrastive Learning: Towards Escaping Uniformity-Tolerance Dilemma in Training

Instance discrimination contrastive learning (CL) has achieved significa...