Virtual Adversarial Training: a Regularization Method for Supervised and Semi-supervised Learning

04/13/2017
by   Takeru Miyato, et al.
0

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the output distribution. Virtual adversarial loss is defined as the robustness of the model's posterior distribution against local perturbation around each input data point. Our method is similar to adversarial training, but differs from adversarial training in that it determines the adversarial direction based only on the output distribution and that it is applicable to a semi-supervised setting. Because the directions in which we smooth the model are virtually adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward and backpropagations. In our experiments, we applied VAT to supervised and semi-supervised learning on multiple benchmark datasets. With additional improvement based on entropy minimization principle, our VAT achieves the state-of-the-art performance on SVHN and CIFAR-10 for semi-supervised learning tasks.

READ FULL TEXT

page 9

page 11

research
07/02/2015

Distributional Smoothing with Virtual Adversarial Training

We propose local distributional smoothness (LDS), a new notion of smooth...
research
11/13/2019

Adversarial Transformations for Semi-Supervised Learning

We propose a Regularization framework based on Adversarial Transformatio...
research
05/25/2016

Adversarial Training Methods for Semi-Supervised Text Classification

Adversarial training provides a means of regularizing supervised learnin...
research
11/12/2019

Negative sampling in semi-supervised learning

We introduce Negative Sampling in Semi-Supervised Learning (NS3L), a sim...
research
09/15/2019

Understanding and Improving Virtual Adversarial Training

In semi-supervised learning, virtual adversarial training (VAT) approach...
research
08/18/2018

Tangent-Normal Adversarial Regularization for Semi-supervised Learning

The ever-increasing size of modern datasets combined with the difficulty...
research
11/26/2020

Regularization with Latent Space Virtual Adversarial Training

Virtual Adversarial Training (VAT) has shown impressive results among re...

Please sign up or login with your details

Forgot password? Click here to reset