Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift

04/06/2019
by   Michał Zając, et al.
0

Recent work has shown that using unlabeled data in semi-supervised learning is not always beneficial and can even hurt generalization, especially when there is a class mismatch between the unlabeled and labeled examples. We investigate this phenomenon for image classification on the CIFAR-10 and the ImageNet datasets, and with many other forms of domain shifts applied (e.g. salt-and-pepper noise). Our main contribution is Split Batch Normalization (Split-BN), a technique to improve SSL when the additional unlabeled data comes from a shifted distribution. We achieve it by using separate batch normalization statistics for unlabeled examples. Due to its simplicity, we recommend it as a standard practice. Finally, we analyse how domain shift affects the SSL training process. In particular, we find that during training the statistics of hidden activations in late layers become markedly different between the unlabeled and the labeled examples.

READ FULL TEXT

page 4

page 11

research
01/16/2019

The information-theoretic value of unlabeled data in semi-supervised learning

We quantify the separation between the numbers of labeled examples requi...
research
07/02/2020

Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning

Existing semi-supervised learning (SSL) algorithms use a single weight t...
research
10/07/2020

Robust Semi-Supervised Learning with Out of Distribution Data

Semi-supervised learning (SSL) based on deep neural networks (DNNs) has ...
research
11/11/2015

Universum Prescription: Regularization using Unlabeled Data

This paper shows that simply prescribing "none of the above" labels to u...
research
10/09/2020

Permuted AdaIN: Enhancing the Representation of Local Cues in Image Classifiers

Recent work has shown that convolutional neural network classifiers over...
research
08/25/2022

Fix-A-Step: Effective Semi-supervised Learning from Uncurated Unlabeled Sets

Semi-supervised learning (SSL) promises gains in accuracy compared to tr...
research
04/07/2021

Streaming Self-Training via Domain-Agnostic Unlabeled Images

We present streaming self-training (SST) that aims to democratize the pr...

Please sign up or login with your details

Forgot password? Click here to reset