ImCLR: Implicit Contrastive Learning for Image Classification

11/25/2020
by   John Chen, et al.
0

Contrastive learning is an effective method for learning visual representations. In most cases, this involves adding an explicit loss function to encourage similar images to have similar representations, and different images to have different representations. Inspired by contrastive learning, we introduce a clever input construction for Implicit Contrastive Learning (ImCLR), primarily in the supervised setting: there, the network can implicitly learn to differentiate between similar and dissimilar images. Each input is presented as a concatenation of two images, and the label is the mean of the two one-hot labels. Furthermore, this requires almost no change to existing pipelines, which allows for easy integration and for fair demonstration of effectiveness on a wide range of well-accepted benchmarks. Namely, there is no change to loss, no change to hyperparameters, and no change to general network architecture. We show that ImCLR improves the test error in the supervised setting across a variety of settings, including 3.24 on CIFAR-100, 0.14 across different number of labeled samples, maintaining approximately a 2 in test accuracy down to using only 5 that gains hold for robustness to common input corruptions and perturbations at varying severities with a 0.72 semi-supervised setting with a 2.16 Π-model. We demonstrate that ImCLR is complementary to existing data augmentation techniques, achieving over 1 improvement on Tiny ImageNet by combining ImCLR with CutMix over either baseline, and 2

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

08/27/2021

Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain

Recent literature in self-supervised has demonstrated significant progre...
01/21/2022

Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentation

Contrastive learning has achieved remarkable success in representation l...
07/02/2021

Supervised Contrastive Learning for Accented Speech Recognition

Neural network based speech recognition systems suffer from performance ...
10/10/2021

Weakly Supervised Contrastive Learning

Unsupervised visual representation learning has gained much attention fr...
05/19/2022

Label-invariant Augmentation for Semi-Supervised Graph Classification

Recently, contrastiveness-based augmentation surges a new climax in the ...
04/15/2022

Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning

An ideal learned representation should display transferability and robus...
07/04/2022

Game State Learning via Game Scene Augmentation

Having access to accurate game state information is of utmost importance...

1 Introduction

Implicit Contrastive Learning (ImCLR). Cross Entropy (CE). All scores are Error %. Generally used most popular implementations on github.

VGG16-CIFAR100 (300 epochs SGD + momentum lr=0.03, mom=0.9. Decay by 0.1 at 150 epochs and 225 epochs. Batch size 128. Augmentation: Center, squish to -1 1, width shift range=0.1, height shift range=0.1, horizontal flip = True)

CE: 27.80 +- .10

CE + CutMix: 27.20 +- .11

CE + ImCLR: 26.50 +- .11

CE + CutMix + ImCLR: 25.49 +- .13

PREACTRN18-CIFAR100 (200 epochs SGD + momentum lr=0.1, mom=0.9. Decay by 0.1 at 60,120,160 epochs. Batch size 128. Augmentation: Center, squish to -1 1, width shift range=0.1, height shift range=0.1, horizontal flip = True)

CE: 25.93 +- .13

CE + ImCLR: 25.29 +- .16

RN20-CIFAR10 (300 epochs SGD + momentum lr=0.08, mom=0.9. Decay by 0.1 at 150 epochs and 225 epochs. Batch size 128. Augmentation: Center, squish to -1 1, width shift range=0.1, height shift range=0.1, horizontal flip = True)

CE: 7.65 +- .12

CE + ImCLR: 7.51 +- .02

WRN16-8-STL10 (100 epochs SGD + momentum lr=0.1, mom=0.9. Decay by 0.1 at 50,75 epochs. Batch size 64. Augmentation: Center, squish to -1 1, width shift range=0.1, height shift range=0.1, horizontal flip = True)

CE: 17.26 +- .08

CE + ImCLR: 14.98 +- .22

VGG16-CIFAR100 ( SGD + momentum lr=0.03, mom=0.9. Decay by 0.1 at 50% epochs and 75% epochs. Batch size 128. Augmentation: Center, squish to -1 1, width shift range=0.1, height shift range=0.1, horizontal flip = True)

% of samples 100 50 30 20 10 5
CE 27.80 .10 34.88 .20 42.52 .34 50.41 .38 71.91 .57 86.03 .12
CE + ImCLR 25.29 .16 33.61 .21 40.40 .34 48.19 .52 68.71 .87 85.61 .40
Table 1: Test error for VGG16-CIFAR100

RN56-Tiny ImageNet (80 epochs SGD + momentum lr=0.1, mom=0.9. Decay by 0.1 at 40 epochs and 60 epochs. Batch size 64. Augmentation: Center, squish to -1 1, width shift range=0.125, height shift range=0.125, horizontal flip = True)

CE : 42.03

CE + ImCLR : 38.79

VGG16-CIFAR100-C (corrupted) (300 epochs SGD + momentum lr=0.03, mom=0.9. Decay by 0.1 at 150 epochs and 225 epochs. Batch size 128. Augmentation: Center, squish to -1 1, width shift range=0.1, height shift range=0.1, horizontal flip = True)

CE: 48.50 +- .13

CE + ImCLR: 47.78

PI model semisupervised learning (cifar10, 450 epochs, decay be 0.2 at 300 epochs. 4000 labels, rest unlabeled. lr=3e-4, coef=20. Flip, random crop, gaussian noise.)

CE: 17.31

CE + ImCLR: 15.15

VGG16-CIFAR100

CE: 27.80 +- .10

CE (ImCLR two stacked): 26.50 +- .11

CE (ImCLR three stacked): 27.35

CE (ImCLR five stacked): 29.25