Big Self-Supervised Models are Strong Semi-Supervised Learners

by   Ting Chen, et al.

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to most previous approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of a big (deep and wide) network during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2 (a modification of SimCLR), supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels (<13 labeled images per class) using ResNet-50, a 10× improvement in label efficiency over the previous state-of-the-art. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels.


page 6

page 15


Estimating Galactic Distances From Images Using Self-supervised Representation Learning

We use a contrastive self-supervised learning framework to estimate dist...

A Studious Approach to Semi-Supervised Learning

The problem of learning from few labeled examples while using large amou...

Streaming Self-Training via Domain-Agnostic Unlabeled Images

We present streaming self-training (SST) that aims to democratize the pr...

Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection

Diabetic retinopathy (DR) is one of the leading causes of blindness in t...

Data-Centric Learning from Unlabeled Graphs with Diffusion Model

Graph property prediction tasks are important and numerous. While each t...

Generate, Annotate, and Learn: Generative Models Advance Self-Training and Knowledge Distillation

Semi-Supervised Learning (SSL) has seen success in many application doma...

An Unsupervised Method for Estimating Class Separability of Datasets with Application to LLMs Fine-Tuning

This paper proposes an unsupervised method that leverages topological ch...

Code Repositories


SimCLRv2 - Big Self-Supervised Models are Strong Semi-Supervised Learners

view repo


Task Agnostic Unsupervised Learning

view repo


use simclr for Kaggle competition

view repo


Unofficial SimCLR multi GPU version

view repo

Please sign up or login with your details

Forgot password? Click here to reset