Improving Consistency-Based Semi-Supervised Learning with Weight Averaging

by   Ben Athiwaratkun, et al.
cornell university

Recent advances in deep unsupervised learning have renewed interest in semi-supervised methods, which can learn from both labeled and unlabeled data. Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. We show that consistency regularization leads to flatter but narrower optima. We also show that the test error surface for these methods is approximately convex in regions of weight space traversed by SGD. Inspired by these observations, we propose to train consistency based semi-supervised models with stochastic weight averaging (SWA), a recent method which averages weights along the trajectory of SGD. We also develop fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With fast-SWA we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100 over many different numbers of observed training labels. For example, we achieve 5.0 the previous best result in the literature. We also improve the best known result from 80 Finally, we show that with fast-SWA the simple Π model becomes state-of-the-art for large labeled settings.


page 1

page 2

page 3

page 4


SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning

This paper introduces SelfMatch, a semi-supervised learning method that ...

AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence

Semi-supervised learning (SSL) is a key approach toward more data-effici...

Milking CowMask for Semi-Supervised Image Classification

Consistency regularization is a technique for semi-supervised learning t...

Virtual Adversarial Ladder Networks For Semi-supervised Learning

Semi-supervised learning (SSL) partially circumvents the high cost of la...

Semi-Supervised Learning with Variational Bayesian Inference and Maximum Uncertainty Regularization

We propose two generic methods for improving semi-supervised learning (S...

Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Size

Advances in Semi-Supervised Learning (SSL) have almost entirely closed t...

Revisiting Consistency Regularization for Semi-Supervised Learning

Consistency regularization is one of the most widely-used techniques for...

Please sign up or login with your details

Forgot password? Click here to reset