Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

03/06/2017
by   Antti Tarvainen, et al.
0

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55 from 35.24

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/12/2021

Couple Learning: Mean Teacher method with pseudo-labels improves semi-supervised deep learning results

The recently proposed Mean Teacher has achieved state-of-the-art results...
research
07/12/2018

Deep semi-supervised segmentation with weight-averaged consistency targets

Recently proposed techniques for semi-supervised learning such as Tempor...
research
06/09/2023

Two Independent Teachers are Better Role Model

Recent deep learning models have attracted substantial attention in infa...
research
11/01/2017

Smooth Neighbors on Teacher Graphs for Semi-supervised Learning

The paper proposes an inductive semi-supervised learning method, called ...
research
08/10/2021

Semi-supervised classification of radiology images with NoTeacher: A Teacher that is not Mean

Deep learning models achieve strong performance for radiology image clas...
research
03/23/2020

Meta Pseudo Labels

Many training algorithms of a deep neural network can be interpreted as ...
research
11/25/2021

Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation

Consistency learning using input image, feature, or network perturbation...

Please sign up or login with your details

Forgot password? Click here to reset