TextAT: Adversarial Training for Natural Language Understanding with Token-Level Perturbation

04/30/2020
by   Linyang Li, et al.
0

Adversarial training is effective in improving the robustness of neural networks. In NLP, languages are discrete in nature, separate tokens possess discrete semantics. Therefore, to incorporate adversarial training in sequence-level tasks, we introduce a novel training strategy: Text Adversarial Training with token-level perturbation. We fist craft perturbations that are initialized using a fine-grained token-level accumulated perturbations. Then we constrain these perturbations considering that inputs are separate tokens, rather than constraining them under a naive normalization ball. We validate the effectiveness of such normalization method using large-scale Transformer-based language models. Experiments on GLUE benchmark and NER task show that our adversarial training strategy improves the performances on various tasks including text classification and sequence labeling.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/25/2019

FreeLB: Enhanced Adversarial Training for Language Understanding

Adversarial training, which minimizes the maximal risk for label-preserv...
research
10/10/2019

Universal Adversarial Perturbation for Text Classification

Given a state-of-the-art deep neural network text classifier, we show th...
research
06/11/2020

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

We present VILLA, the first known effort on large-scale adversarial trai...
research
04/12/2021

Targeted Adversarial Training for Natural Language Understanding

We present a simple yet effective Targeted Adversarial Training (TAT) al...
research
04/15/2021

Consistency Training with Virtual Adversarial Discrete Perturbation

We propose an effective consistency training framework that enforces a t...
research
09/15/2021

ARCH: Efficient Adversarial Regularized Training with Caching

Adversarial regularization can improve model generalization in many natu...
research
04/13/2022

A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweets

Health mention classification deals with the disease detection in a give...

Please sign up or login with your details

Forgot password? Click here to reset