Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation

by   Chong Zhang, et al.

Robustness and counterfactual bias are usually evaluated on a test dataset. However, are these evaluations robust? If the test dataset is perturbed slightly, will the evaluation results keep the same? In this paper, we propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. The framework first perturbs the test dataset to construct abundant natural sentences similar to the test data, and then diagnoses the prediction change regarding a single-word substitution. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English. (1) For robustness, we focus on synonym substitutions and identify vulnerable examples where prediction can be altered. Our proposed attack attains high success rates (96.0 vulnerable examples on both original and robustly trained CNNs and Transformers. (2) For counterfactual bias, we focus on substituting demographic tokens (e.g., gender, race) and measure the shift of the expected prediction among constructed sentences. Our method is able to reveal the hidden model biases not directly shown in the test dataset. Our code is available at


page 2

page 7

page 17

page 18


Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures

Existing NLP datasets contain various biases, and models tend to quickly...

Towards Robustifying NLI Models Against Lexical Dataset Biases

While deep learning models are making fast progress on the task of Natur...

Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models

Vision-Language Pre-training (VLP) models have achieved state-of-the-art...

ISCAS at SemEval-2020 Task 5: Pre-trained Transformers for Counterfactual Statement Modeling

ISCAS participated in two subtasks of SemEval 2020 Task 5: detecting cou...

Color Channel Perturbation Attacks for Fooling Convolutional Neural Networks and A Defense Against Such Attacks

The Convolutional Neural Networks (CNNs) have emerged as a very powerful...

Balancing out Bias: Achieving Fairness Through Training Reweighting

Bias in natural language processing arises primarily from models learnin...

Gendered Ambiguous Pronouns Shared Task: Boosting Model Confidence by Evidence Pooling

This paper presents a strong set of results for resolving gendered ambig...