BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classification

09/15/2021
by   Jens Hauser, et al.
0

Deep Neural Networks have taken Natural Language Processing by storm. While this led to incredible improvements across many tasks, it also initiated a new research field, questioning the robustness of these neural networks by attacking them. In this paper, we investigate four word substitution-based attacks on BERT. We combine a human evaluation of individual word substitutions and a probabilistic analysis to show that between 96 attacks do not preserve semantics, indicating that their success is mainly based on feeding poor data to the model. To further confirm that, we introduce an efficient data augmentation procedure and show that many adversarial examples can be prevented by including data similar to the attacks during training. An additional post-processing step reduces the success rates of state-of-the-art attacks below 5 thresholds on constraints for word substitutions, we conclude that BERT is a lot more robust than research on attacks suggests.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/27/2019

Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment

Machine learning algorithms are often vulnerable to adversarial examples...
research
04/25/2020

Reevaluating Adversarial Examples in Natural Language

State-of-the-art attacks on NLP models have different definitions of wha...
research
12/24/2020

A Context Aware Approach for Generating Natural Language Attacks

We study an important task of attacking natural language processing mode...
research
02/27/2020

Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT

There is an increasing amount of literature that claims the brittleness ...
research
11/19/2019

Towards non-toxic landscapes: Automatic toxic comment detection using DNN

The spectacular expansion of the Internet led to the development of a ne...
research
05/24/2022

Certified Robustness Against Natural Language Attacks by Causal Intervention

Deep learning models have achieved great success in many fields, yet the...
research
01/10/2021

BERT Family Eat Word Salad: Experiments with Text Understanding

In this paper, we study the response of large models from the BERT famil...

Please sign up or login with your details

Forgot password? Click here to reset