Meta Pseudo Labels

03/23/2020
by   Hieu Pham, et al.
0

Many training algorithms of a deep neural network can be interpreted as minimizing the cross entropy loss between the prediction made by the network and a target distribution. In supervised learning, this target distribution is typically the ground-truth one-hot vector. In semi-supervised learning, this target distribution is typically generated by a pre-trained teacher model to train the main network. In this work, instead of using such predefined target distributions, we show that learning to adjust the target distribution based on the learning state of the main network can lead to better performances. In particular, we propose an efficient meta-learning algorithm, which encourages the teacher to adjust the target distributions of training examples in the manner that improves the learning of the main network. The teacher is updated by policy gradients computed by evaluating the main network on a held-out validation set. Our experiments demonstrate substantial improvements over strong baselines and establish state-ofthe-art performance on CIFAR-10, SVHN, and ImageNet. For instance, with ResNets on small datasets, we achieve 96.1 4,000 labeled examples and 73.9 Meanwhile, with EfficientNet on full datasets plus extra unlabeled data, we attain 98.6

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/03/2022

Semi-supervised Learning using Robust Loss

The amount of manually labeled data is limited in medical applications, ...
research
03/03/2021

Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

While recent studies on semi-supervised learning have shown remarkable p...
research
12/27/2022

Self Meta Pseudo Labels: Meta Pseudo Labels Without The Teacher

We present Self Meta Pseudo Labels, a novel semi-supervised learning met...
research
07/05/2020

Meta-Semi: A Meta-learning Approach for Semi-supervised Learning

Deep learning based semi-supervised learning (SSL) algorithms have led t...
research
06/06/2019

What you need is a more professional teacher

We propose a simple and efficient method to combine semi-supervised lear...
research
11/21/2019

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

We improve the recently-proposed "MixMatch" semi-supervised learning alg...
research
03/06/2017

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

The recently proposed Temporal Ensembling has achieved state-of-the-art ...

Please sign up or login with your details

Forgot password? Click here to reset