Self-training with Noisy Student improves ImageNet classification

11/11/2019
by   Qizhe Xie, et al.
36

We present a simple self-training method that achieves 87.4 on ImageNet, which is 1.0 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 16.6 corruption error from 45.7 to 31.2, and reduces ImageNet-P mean flip rate from 27.8 to 16.1. To achieve this result, we first train an EfficientNet model on labeled ImageNet images and use it as a teacher to generate pseudo labels on 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the generation of the pseudo labels, the teacher is not noised so that the pseudo labels are as good as possible. But during the learning of the student, we inject noise such as data augmentation, dropout, stochastic depth to the student so that the noised student is forced to learn harder from the pseudo labels.

READ FULL TEXT
research
06/03/2021

Noisy student-teacher training for robust keyword spotting

We propose self-training with noisy student-teacher approach for streami...
research
10/15/2020

Unsupervised Self-training Algorithm Based on Deep Learning for Optical Aerial Images Change Detection

Optical aerial images change detection is an important task in earth obs...
research
08/25/2021

Multi-Task Self-Training for Learning General Representations

Despite the fast progress in training specialized models for various tas...
research
01/06/2022

Self-Training Vision Language BERTs with a Unified Conditional Model

Natural language BERTs are trained with language corpus in a self-superv...
research
11/20/2020

SLADE: A Self-Training Framework For Distance Metric Learning

Most existing distance metric learning approaches use fully labeled data...
research
09/15/2021

Self-Training with Differentiable Teacher

Self-training achieves enormous success in various semi-supervised and w...
research
07/19/2020

Self-similarity Student for Partial Label Histopathology Image Segmentation

Delineation of cancerous regions in gigapixel whole slide images (WSIs) ...

Please sign up or login with your details

Forgot password? Click here to reset