PyTorch implementation of MoCo: https://arxiv.org/abs/1911.05722
Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo—namely, using an MLP projection head and more data augmentation—we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.READ FULL TEXT VIEW PDF
PyTorch implementation of MoCo: https://arxiv.org/abs/1911.05722
Video Noise Contrastive Estimation
TF 2.x implementation of MoCo v1 (Momentum Contrast for Unsupervised Visual Representation Learning, CVPR 2020) and MoCo v2 (Improved Baselines with Momentum Contrastive Learning, 2020).
Code accompagning the paper: Integration Categorical Semantics into Unsupervised Domain Translation: https://arxiv.org/abs/2010.01262.
Recent studies on unsupervised representation learning from images [16, 13, 8, 17, 1, 9, 15, 6, 12, 2] are converging on a central concept known as contrastive learning . The results are promising: e.g., Momentum Contrast (MoCo) 
shows that unsupervised pre-training can surpass its ImageNet-supervised counterpart in multiple detection and segmentation tasks, and SimCLR
further reduces the gap in linear classifier performance between unsupervised and supervised pre-training representations.
This note establishes stronger and more feasible baselines built in the MoCo framework. We report that two design improvements used in SimCLR, namely, an MLP projection head and stronger data augmentation, are orthogonal to the frameworks of MoCo and SimCLR, and when used with MoCo they lead to better image classification and object detection transfer learning results. Moreover, the MoCo framework can process a large set of negative samples without requiring large training batches (Fig.1). In contrast to SimCLR’s large 4k8k batches, which require TPU support, our “MoCo v2” baselines can run on a typical 8-GPU machine and achieve better results than SimCLR. We hope these improved baselines will provide a reference for future research in unsupervised learning.
Contrastive learning 
is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a dictionary look-up problem. An effective contrastive loss function, called InfoNCE, is:
Here is a query representation, is a representation of the positive (similar) key sample, and are representations of the negative (dissimilar) key samples. is a temperature hyper-parameter. In the instance discrimination pretext task  (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.
The contrastive loss (1) can be minimized by various mechanisms that differ in how the keys are maintained . In an end-to-end mechanism (Fig. 1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end-to-end by back-propagation. SimCLR  is based on this mechanism and requires a large batch to provide a large set of negatives. In the MoCo mechanism (Fig. 1b) , the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch. A momentum encoder is adopted to improve the representation consistency between the current and earlier keys. MoCo decouples the batch size from the number of negatives.
SimCLR  improves the end-to-end variant of instance discrimination in three aspects: (i) a substantially larger batch (4k or 8k) that can provide more negative samples; (ii) replacing the output fc projection head  with an MLP head; (iii) stronger data augmentation.
In the MoCo framework, a large number of negative samples are readily available; the MLP head and data augmentation are orthogonal to how contrastive learning is instantiated. Next we study these improvements in MoCo.
|unsup. pre-train||ImageNet||VOC detection|
Unsupervised learning is conducted on the 1.28M ImageNet  training set. We follow two common protocols for evaluation. (i) ImageNet linear classification: features are frozen and a supervised linear classifier is trained; we report 1-crop (224224), top-1 validation accuracy. (ii) Transferring to VOC object detection : a Faster R-CNN detector  (C4-backbone) is fine-tuned end-to-end on the VOC 07+12 trainval set111For all entries (including the supervised and MoCo v1 baselines), we fine-tune for 24k iterations on VOC, up from 18k in . and evaluated on the VOC 07 test set using the COCO suite of metrics . We use the same hyper-parameters (except when noted) and codebase as MoCo . All results use a standard-size ResNet-50 .
Following , we replace the fc
head in MoCo with a 2-layer MLP head (hidden layer 2048-d, with ReLU). Note this only influences the unsupervised training stage; thelinear classification or transferring stage does not use this MLP head. Also, following , we search for an optimal w.r.t. ImageNet linear classification accuracy:
Using the default 0.07 [16, 6], pre-training with the MLP head improves from 60.6% to 62.9%; switching to the optimal value for MLP (0.2), the accuracy increases to 66.2%. Table 1(a) shows its detection results: in contrast to the big leap on ImageNet, the detection gains are smaller.
We extend the original augmentation in  by including the blur augmentation in  (we find the stronger color distortion in  has diminishing gains in our higher baselines). The extra augmentation alone (i.e., no MLP) improves the MoCo baseline on ImageNet by 2.8% to 63.4%, Table 1(b). Interestingly, its detection accuracy is higher than that of using the MLP alone, Table 1(b) vs. (a), despite much lower linear classification accuracy (63.4% vs. 66.2%). This indicates that linear classification accuracy is not monotonically related to transfer performance in detection. With the MLP, the extra augmentation boosts ImageNet accuracy to 67.3%, Table 1(c).
Table 2 compares SimCLR  with our results, referred to as MoCo v2. For fair comparisons, we also study a cosine (half-period) learning rate schedule  which SimCLR adopts. See Table 1(d, e). Using pre-training with 200 epochs and a batch size of 256, MoCo v2 achieves 67.5% accuracy on ImageNet: this is 5.6% higher than SimCLR under the same epochs and batch size, and better than SimCLR’s large-batch result 66.6%. With 800-epoch pre-training, MoCo v2 achieves 71.1%, outperforming SimCLR’s 69.3% with 1000 epochs.
|MoCo v1 ||200||256||60.6|
|results of longer unsupervised training follow:|
In Table 3 we report the memory and time cost of our implementation. The end-to-end case reflects the SimCLR cost in GPUs (instead of TPUs in ). The 4k batch size is intractable even in a high-end 8-GPU machine. Also, under the same batch size of 256, the end-to-end variant is still more costly in memory and time, because it back-propagates to both and encoders, while MoCo back-propagates to the encoder only.
SGDR: Stochastic gradient descent with warm restarts.In ICLR, 2017.