Adversarial Dropout for Recurrent Neural Networks

04/22/2019
by   Sungrae Park, et al.
0

Successful application processing sequential data, such as text and speech, requires an improved generalization performance of recurrent neural networks (RNNs). Dropout techniques for RNNs were introduced to respond to these demands, but we conjecture that the dropout on RNNs could have been improved by adopting the adversarial concept. This paper investigates ways to improve the dropout for RNNs by utilizing intentionally generated dropout masks. Specifically, the guided dropout used in this research is called as adversarial dropout, which adversarially disconnects neurons that are dominantly used to predict correct targets over time. Our analysis showed that our regularizer, which consists of a gap between the original and the reconfigured RNNs, was the upper bound of the gap between the training and the inference phases of the random dropout. We demonstrated that minimizing our regularizer improved the effectiveness of the dropout for RNNs on sequential MNIST tasks, semi-supervised text classification tasks, and language modeling tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/31/2017

Fraternal Dropout

Recurrent neural networks (RNNs) are important class of architectures am...
research
11/04/2013

On Fast Dropout and its Applicability to Recurrent Networks

Recurrent Neural Networks (RNNs) are rich models for the processing of s...
research
07/12/2017

Adversarial Dropout for Supervised and Semi-supervised Learning

Recently, the training with adversarial examples, which are generated by...
research
05/29/2018

Deep Learning under Privileged Information Using Heteroscedastic Dropout

Unlike machines, humans learn through rapid, abstract model-building. Th...
research
11/19/2019

Thick-Net: Parallel Network Structure for Sequential Modeling

Recurrent neural networks have been widely used in sequence learning tas...
research
03/10/2020

Prediction of Bayesian Intervals for Tropical Storms

Building on recent research for prediction of hurricane trajectories usi...
research
10/21/2014

Regularizing Recurrent Networks - On Injected Noise and Norm-based Methods

Advancements in parallel processing have lead to a surge in multilayer p...

Please sign up or login with your details

Forgot password? Click here to reset