Semantically-Conditioned Negative Samples for Efficient Contrastive Learning

02/12/2021
by   James O'Neill, et al.
26

Negative sampling is a limiting factor w.r.t. the generalization of metric-learned neural networks. We show that uniform negative sampling provides little information about the class boundaries and thus propose three novel techniques for efficient negative sampling: drawing negative samples from (1) the top-k most semantically similar classes, (2) the top-k most semantically similar samples and (3) interpolating between contrastive latent representations to create pseudo negatives. Our experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet-200 show that our proposed Semantically Conditioned Negative Sampling and Latent Mixup lead to consistent performance improvements. In the standard supervised learning setting, on average we increase test accuracy by 1.52% percentage points on CIFAR-10 across various network architectures. In the knowledge distillation setting, (1) the performance of student networks increase by 4.56% percentage points on Tiny-ImageNet-200 and 3.29% on CIFAR-100 over student networks trained with no teacher and (2) 1.23% and 1.72% respectively over a hard-to-beat baseline (Hinton et al., 2015).

READ FULL TEXT

page 7

page 17

page 18

page 19

page 20

research
05/03/2022

Do More Negative Samples Necessarily Hurt in Contrastive Learning?

Recent investigations in noise contrastive estimation suggest, both empi...
research
02/25/2019

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Recent empirical works have successfully used unlabeled data to learn fe...
research
10/09/2020

Contrastive Learning with Hard Negative Samples

We consider the question: how can you sample good negative examples for ...
research
12/25/2022

BD-KD: Balancing the Divergences for Online Knowledge Distillation

Knowledge distillation (KD) has gained a lot of attention in the field o...
research
08/29/2021

Lipschitz Continuity Guided Knowledge Distillation

Knowledge distillation has become one of the most important model compre...
research
10/02/2018

CINIC-10 is not ImageNet or CIFAR-10

In this brief technical report we introduce the CINIC-10 dataset as a pl...

Please sign up or login with your details

Forgot password? Click here to reset