Semantically-Conditioned Negative Samples for Efficient Contrastive Learning

02/12/2021 ∙ by James O'Neill, et al. ∙ 26

Negative sampling is a limiting factor w.r.t. the generalization of metric-learned neural networks. We show that uniform negative sampling provides little information about the class boundaries and thus propose three novel techniques for efficient negative sampling: drawing negative samples from (1) the top-k most semantically similar classes, (2) the top-k most semantically similar samples and (3) interpolating between contrastive latent representations to create pseudo negatives. Our experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet-200 show that our proposed Semantically Conditioned Negative Sampling and Latent Mixup lead to consistent performance improvements. In the standard supervised learning setting, on average we increase test accuracy by 1.52% percentage points on CIFAR-10 across various network architectures. In the knowledge distillation setting, (1) the performance of student networks increase by 4.56% percentage points on Tiny-ImageNet-200 and 3.29% on CIFAR-100 over student networks trained with no teacher and (2) 1.23% and 1.72% respectively over a hard-to-beat baseline (Hinton et al., 2015).



There are no comments yet.


page 7

page 17

page 18

page 19

page 20

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.