Syntactic Data Augmentation Increases Robustness to Inference Heuristics

04/24/2020
by   Junghyun Min, et al.
0

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model's limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the fine-tuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT's accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/31/2021

How Does Adversarial Fine-Tuning Benefit BERT?

Adversarial training (AT) is one of the most reliable methods for defend...
research
04/04/2023

Calibrated Chaos: Variance Between Runs of Neural Network Training is Harmless and Inevitable

Typical neural network trainings have substantial variance in test-set p...
research
04/30/2020

Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs

Sequence-based neural networks show significant sensitivity to syntactic...
research
11/07/2019

BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance

If the same neural architecture is trained multiple times on the same da...
research
02/04/2019

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

Machine learning systems can often achieve high performance on a test se...
research
11/04/2020

Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalization

Previous studies investigating the syntactic abilities of deep learning ...
research
06/01/2022

Order-sensitive Shapley Values for Evaluating Conceptual Soundness of NLP Models

Previous works show that deep NLP models are not always conceptually sou...

Please sign up or login with your details

Forgot password? Click here to reset