Significant progress for a large number of natural language processing tasks has been made through the development of new deep neural models; higher scores for shared tasks such as Natural Language Inference and Question Answering  have been achieved through models which are becoming increasingly complex. This complexity raises new challenges: as models become more complex, it becomes difficult to fully understand and characterize their behaviour. This has implications from a statistical learning perspective as the more complex models can be prone to brittleness. From an NLP perspective, there has been an ongoing discussion as to what extent these models understand language  or to what extent they are exploiting unintentional biases and cues that are present in the datasets they are trained on [22, 9]. When the model is evaluated on data outside of the distribution defined (implicitly) by its training dataset, its behaviour is likely to be unpredictable; such “blind spots” can be exposed through adversarial evaluation .
In this paper we focus on the use of adversarial evaluation in the context of fact checking: this is the task of predicting whether a claim is supported or refuted by evidence (see  for an in-depth task description). As automated systems for fact checking have potentially sensitive applications it is important to study the vulnerabilities of these systems, as well as the deficiencies of the datasets they are trained on. Through adversarial evaluation, where systems are exposed to examples of statements intentionally created to result in incorrect predictions, we can better understand the limitations of the systems and possibly use these adversarial instances to regularize the model through training data augmentation [26, 8, 17].
The Fact Extraction and VERification (FEVER) shared task  invited participants to build systems to predict whether short factoid sentences (referred to as claims) – manually written using information derived from Wikipedia – are Supported or Refuted by evidence or whether there is NotEnoughInfo to reach a conclusion. Most participants in the shared task modeled it as pipeline consisting of an information retrieval component – finding pages/sentences with content relevant to a given claim – and a Natural Language Inference (NLI) component – proving that given the evidence, a claim is either Supported or Refuted. The second iteration of the task (FEVER2.0) is building on the same dataset, but takes the form of a build-it, break-it, fix-it task . Builders will create systems based on the original FEVER dataset and task definition; breakers will generate adversarial examples targeting the systems built in the first stage; finally, fixers will implement solutions to remedy the attacks from the second stage.
|FEVER Claim: Bullitt is a movie directed by Phillip D’Antoni|
|Evidence: Bullitt is a 1968 American action thriller film directed by Peter Yates and produced by Philip D’Antoni|
|Entailment Preserving Transformation||Entailment Altering Transformation|
|There is a movie directed by Phillip D’Antoni, it is called Bullitt.||Bullitt is not a movie directed by Phillip D’Antoni|
|New Label: Refuted||New Label: Supported|
We describe a baseline for the breaking stage FEVER2.0 shared task: we use simple, rule-based transformations of claims from the original FEVER dataset to generate new ones from existing instances in the dataset. This approach makes use of the same evidence, avoiding the need for additional evidence finding, and the rules can either change or preserve their label with respect to the evidence (see Figure 1). We also introduce metrics for scoring the potency
of the attacks (which accounts for the number of systems that incorrectly classify the newly generated adversarial instances). We attack a number of state-of-the-art models: the Neural Semantic Matching Network, HexaF Model , Athene  and Papelo  – the highest ranked systems for the first FEVER shared task – as well as an ESIM model  with ELMo embeddings 
and a Decomposable Attention model – built on the baseline architecture presented in . We evaluated the effect of the rule-based transformations on both the Information Retrieval and NLI components of these systems and found that all systems achieved lower accuracyon the new data we generated with decreases of between and . Also, while the information retrieval components of some systems were not affected to the same extent, our experimental results indicate that some implementations were more sensitive the new, adversarial, data than others. We hope that this baseline implementation and report serves as a useful resource for future participants of the FEVER2.0 shared task by showing that a relatively simple method reasonably effective and highlighting in what ways the models we test against are affected. To help participants, we release dockerized versions of the systems we test against 111https://github.com/j6mes/fever-docker as well as code for generating and evaluating adversarial examples 222https://github.com/j6mes/fever2-baseline.
2 Related Works
Adversarial examples have been initially studied in the field of computer vision.Szegedy et al.  identified an over-sensitivity of state-of-the-art models whereby perturbations to the image (by altering pixel intensities in a way which is imperceptible to humans) resulted in changes to the predictions made by the model. These perturbation were generated through altering the model input to maximize the classification error. While the proposed methods of altering pixel intensities was generally imperceptible to humans, making similar perturbations to text is more challenging due to the discrete symbol space and the need to preserve grammaticality: modifying a single token may either change the label of the instance or introduce grammatical errors. Various methods for successful attacks have been proposed, including manual construction, character perturbations, addition of distractor information, rule-based transformations and paraphrasing, and automated generation. In our brief survey, we compare these methods and the trade-off between the level of automation (which allows both scale and diversity of new claims) and whether the perturbation unintentionally changes the label of the instance or induces grammatical error, which would require human annotation to identify and resolve.
Character-level attacks have also highlighted the brittleness of NLP systems: by making letter swaps or insertions, Belinkov and Bisk  and Ebrahimi et al.  have generated distorted examples which cause misclassifications or translation errors. While it is unlikely that a single character can unintentionally change the meaning of a sentence, this method is still intentionally introducing errors.
Jia and Liang  evaluated the addition of distractor information in reading comprehension systems. Adversarial instances are generated for the SQuAD  shared task (question answering against a short passage of text) by concatenating short distractor sentences to the passage. The distractor sentences are generated through perturbing the question with entity substitution and generating a false answer which has a similar form to the actual answer with rule-based substitutions. Furthermore, the additional information concatenated to the original passage of text is by construction irrelevant, as it is about another entity. Thus it is unlikely to cause a change to the meaning of the text requiring the instance to be labelled. Additionally, this approach does not require manual generation of the instances (human annotators are only used for filtering out ungrammatical distractors) this would be less expensive than manual construction of adversarial instances meaning that greater scale can be achieved without the risk of annotators unintentionally introducing a bias.
Iyyer et al.  and Ribeiro et al.  apply paraphrase-based transformations to generate adversarial instances using alignments from parallel corpora from translation tasks. Iyyer et al.  attack sentiment analysis and recognizing textual entailment systems, generating paraphrases of instances with a encoder-decoder model architecture. In the process of generating adversarial instances, the meaning could be altered requiring relabelling (for example a sentence pair with an Entailment relation may become Neutral
after perturbation) or the newly generated text could be ungrammatical. In an error analysis, the authors identified that in 17.7-22.3% of cases, the generated examples were not paraphrases and in 14.0-19.3% of cases, the paraphrases were ungrammatical.Ribeiro et al. 
evaluate phrase substitutions, obtained from alignments from a translation task, as a method of generating semantically equivalent instances for visual QA, sentiment analysis, and reading comprehension. The authors incorporate additional filtering to remove instances that are ungrammatical or unnatural: rather than using human annotators, this is automated through computing the probability of translating the original instance and getting the paraphrase when back-translating.
Programmatic Construction of Adversarial Dataset
Naik et al.  introduce a stress test evaluation dataset for NLI containing a number of methods for generating new claims that exploit limitations and biases present in state-of-the-art models in the context of the MultiNLI  shared task. The adversarial dataset was constructed by applying three types of transformations to the MultiNLI development data split: meaning-altering transformations are performed to instances that require numerical reasoning through rule-based transformation; distractor phrases that preserve meaning are appended to instances (that exploit models’ biases for strong indicators for negation and sentence length); finally, perturbations to some instances are introduced to mimic typographical errors. While the rule-based changes would preserve the label and are grammatical, some of the changes in this dataset are not natural (for example, the word overlap rule which appended ‘and true is true’ tautology is logically correct but unlikely to occur in everyday language). This approach is relatively cheap in comparison to manual construction as one rule can be applied to many instances.
Zhao et al. 
generate natural language adversaries for an NLI task through the use of an autoencoder architecture. The new instances exhibit high diversity and are regarded as “natural” with 86% of human annotators stating that the new instances were grammatical and 81% stating that the new instances was similar to the original on a pilot study of 20 examples. However, while this method will generate instances that are similar to the original, it is not certain that the label for the newly generated instances will be preserved as similarity does not guarantee semantic equivalence.Zellers et al.  introduce an adversarial method for generating negative examples for a multiple-choice answer selection task through generating probable sentences (with the aid of a language model) that induce misclassifications. Using this approach, however, it is not possible to generate examples that preserve or deterministically alter the entailment relation: again requiring the newly generated instances to be labeled by humans.
3 Adversarial Attacks Against FEVER
(The Break-it Phase of the FEVER2.0 Shared Task)
|FEVER Claim: Lily James has been on TV.|
|Evidence Combination 1||Evidence Combination 2|
|[wiki/Lily_James:1] She studied acting at the Guildhall School of Music and Drama in London and began her acting career in the British television series Just William (2010)||[wiki/Lily_James:0] Following her supporting role as Lady Rose MacClare in the period drama series Downton Abbey (2012–2015), James had her film breakthrough playing the titular role in the fantasy film Cinderella (2015).|
|[wiki/Downton_Abbey:0] Downton Abbey is a British historical period drama television series set in the early 20th century, created by Julian Fellowes.|
3.1 Task Definition
We start by formally defining the FEVER task for which we will be generating adversarial instances for. A prediction for a FEVER instance (example in Figure 2) that comprises a claim sentence , correct (gold) label , predicted label , set of correct evidence sentence combinations and predicted evidence sentence combination , it is scored as using equations 1 and 2.
For claims labeled NotEnoughInfo, no evidence is required. Some claims require the combination of multiple sentences to be marked as fully supported or refuted: all sentences must be present in order for the submission to be marked as correct. As there may be multiple correct combinations of evidence sentences that can be used to Support or Refute a claim, at least one of these combinations needs to be predicted in its entirety for the prediction to be considered correct. 333The full FEVER scorer is available as a stand-alone package on GitHub: https://github.com/sheffieldnlp/fever-scorer and can be installed with pip install fever-scorer
Over a set of predictions made on labeled instances, , the FEVER score is defined as:
For diagnostic purposes, we also report label accuracy (ignoring the need for correct evidence):
The FEVER2.0 shared task requires breakers (participants generating adversarial instances) , to generate and submit a balanced set (i.e. same number of instances per class) of novel claims , a label and set of evidence sentence combinations . The number of novel claims in the submission will be limited: this will allow for manual evaluation to ensure that each submitted adversarial instance meets the shared task guidelines444An up-to-date version of the guidelines are available on the FEVER website: http://fever.ai/task.html. The adversarial instances generated that meet the guidelines (hereafter referred to as accepted instances: , ) are then submitted to all systems from the builders, and the predictions made by these systems are used for scoring.
We evaluate the adversarial instances from each breaker (ignoring the ones that failed to meet the guidelines) in two ways. The potency of the adversarial instances submitted by a breaker is defined as the average error rate over all the of predictions made by all systems for labeled accepted instances generated by a breaker , :
To penalize breakers for submitting claims that do not meet the guidelines (such as those which are ungrammatical or incorrectly labeled), the potency score will be scaled by the acceptance rate () of the claims submitted:
The resilience of a system is defined as the FEVER score over all the accepted instances generated by all the breakers:
The use of human annotators to construct large scale datasets for natural language processing risks inducing biases and artifacts as the workers adopt strategies and shortcuts to generate examples. We hypothesize that the annotators for FEVER used a common set of constructions that could be exploited to generate a new set of adversarial claims. Counting the bigrams within the claims in training set, we identify the following forms: "is a", "an American", "was born", "directed by", "starred in" are some of the most frequently occurring.
Using these common claim patterns, we apply a collection of rule based transformations to the claims to generate new instances that do not reflect the patterns available in the training set. We evaluate three different groups of transformations: one where entailment between the claim and evidence is preserved, and two where it is reversed through negation (see examples in Table 1). The first set of transformations are entailment-preserving rewrites of the claim. There are similar to the methods that the annotators would have used when generating the FEVER dataset and includes simple techniques such as switching from active to passive voice, and adding pronouns. The second set of transformations were simple negations to the claim, performed mostly through a simple negation the sentences’ verb phrase and reversing any Supported label to Refuted and vice versa. For some claim types, intensifiers such as ‘certainly’ and ‘definitely’ were also included in the sentence where this would not affect the entailment relation (as defined in Appendix A of ). The third set of rules combine both the entailment-altering rewrites and the simple negations (without the inclusion of any intensifiers), and we refer to them as complex negations.
We constructed 65 rules that matched the most common claim patterns, for example: is a , was an , was a , was an , was directed by , died on , died in , was born in , is an American . 23 of these rules were entailment preserving, 19 were simple changes to negate claims, 23 were complex changes that also negated the claims. We release all of the rules that were used in this paper on our GitHub repository.
|Entailment Preserving||(.+) is a (.+)||There exists a $2 called $1|
|(.+) (?:was | is)? directed by (.+)||$2 is the director of $1|
|Simple Negation||(.+) was an (.+)||$1 was not an $2|
|(.+) was born in (.+)||$1 was never born|
|Complex Negation||(.+) (?:was | is)? directed by (.+)||There is a movie called $1 which wasn’t directed by $2|
|(.+) an American (.+)||$1 $2 that originated from outside the United States.|
4 Experimental Setup
We evaluate pretrained models for the FEVER shared task and their robustness to the adversarial instances we introduce in this paper. In this section, we compare the baseline model from Thorne et al.  (TF-IDF information retrieval with a decomposable attention model for natural language inference), a modified version of this model with an ESIM+ELMo for the NLI component, and the top four highest performing scoring system from the 2018 task [19, 31, 10, 16]. For the baseline system, we adapted the code from , using the ESIM+ELMo implementation from AllenNLP Gardner et al.  and trained both the Decomposable Attention and ESIM+ELMo models with the recommended parameters for each model as included in the AllenNLP implementation. For the four highest scoring systems from the first shared task, we used the pretrained models released by each of the participants.
The adversarial claims are generated by applying rule-based transformations (described in Section 3.2) to existing FEVER instances. In this paper, we made observations on the FEVER training set, counting common bigrams, that informed the creation of transformation rules that when applied to the dataset would create a large number of instances. In our experiments, we chose to retain the original evidence and only make changes to the claim text because it mitigates the need to expend effort annotating new evidence. We apply these rules to both the FEVER development and test splits of the dataset split presented in Thorne et al.  555Note: These are the development split and test split from the paper containing examples each that were later concatenated, forming the development set of the first FEVER shared task to generate a two datasets of novel claims. In Section 5, we evaluate the effects of the rule-base adversary using the claims from the development split and reporting the impact on both the NLI and Evidence Retrieval components. We report changes in performance over the entire pipeline as well as in an oracle environment where only the NLI component is tested. For this oracle evaluation, we replace the evidence retrieval module with correct evidence as labeled by the annotators. For claims labeled as NotEnoughInfo in the oracle, we sample sentences from the nearest Wikipedia page to the claim (using TF-IDF, consistent with the NearestP approach reported by Thorne et al. ). We report the changes in FEVER Score over the end-to-end pipeline system, as well as report the changes in accuracy of the NLI model and the precision, recall and of the evidence retrieval component. In Section 6, we create a sample submission for the FEVER2.0 shared task and report the potency of our technique and the resilience of the systems we evaluate against.
65 rule-based transformations were executed to the data from the development set. There were instances that matched the regular expression patterns defined in these rules, which were used to generate novel adversarial instances. We report results using the instances which matched the rules (column labeled Before in Table 2) that were used to generate the novel adversarial instances (column labeled After). In the new set of claims that were generated, only were labeled as NotEnoughInfo (as the negating rules were only applied to claims which were Supported or Refuted) which would preclude comparison against existing results as the FEVER evaluation which assumes balanced class distribution. We balance our data by discarding instances at random from the majority classes. In a more comprehensive breakdown in Section 5.1, we report results on all instances, broken down by the types of rules used in the adversarial attack.
|Oracle + DA||82.83||87.94||68.06||-19.88||=||=||=||=|
|Oracle + ESIM||84.63||86.93||66.87||-19.16||=||=||=||=|
|TFIDF + DA||51.94||50.50||39.18||-11.32||28.63||26.88||18.67||-8.21|
|TFIDF + ESIM||54.01||51.30||39.89||-11.41||33.72||33.17||21.62||-11.55|
We observe that in the oracle environment (using manually labeled evidence), both the ESIM and Decomposable Attention models for natural language inference suffer a stark decrease in accuracy when making predictions on the adversarial examples. The decrease is less pronounced when considering the full pipeline (i.e. with a non-perfect evidence retrieval component), which reduces the upper bound for the FEVER Score and accuracy. This is due to the noise introduced by the evidence retrieval component documented in Table 3.
The adversarial examples did not affect the information retrieval component of some systems to the same extent as the NLI component. Most systems incorporated either TF-IDF of keyword matching in their information retrieval component and thus they are little affected by our rule-based transformations which adding mostly stop words and reordering the words within the sentence. The only expecption is Papelo, which even though uses TF-IDF to retrieve documents, its sentence-retrieval component made use of an entailment classifier. While this approach maintained very high precision with the newly generated instances, the recall decreased indicating a brittleness for instances that that differed from the distribution of the training set.
5.1 Effect of Transformation Types
In Section 3.2, we introduced three types of rule-based transformations (entailment-preserving paraphrasing, simple negations, and complex negations that combine the first two) that were used to generate the adversarial instances. Each of these rule-based transformations may expose different blind-spots and limitations within the models: we evaluate these in isolation and present a summary in Table 4 where we show that the systems under consideration exhibited a reduction in accuracy for each our rule-based transformation types.
We observe in all oracle and pipeline cases (with the exception of the Athene system) that the reduction in accuracy for the claims generated through label-preserving transformations was lower than the claims generated by the label-altering transformations. The effect of this is also exacerbated by the fact that the negated claims is only a binary classification rather than three-way for the entailment preserving claims. This may be revealing an inherent bias in the models similar to the one discussed by Naik et al. , where models perform poorly for antonymous examples due to a dependence on word-overlap as a feature, which we also observe with our negations. We also observe that 3 of top 4 ranking systems from the shared task (UNC, UCL and Papelo) exhibit a reduction in FEVER scorer for the negated instances. For the baseline (TFIDF+DA/ESIM) models, simple negations had a lower impact on the FEVER score than entailment preserving claims, possibly due to the noise coming from the poor evidence retrieval method (under oracle evaluation, this was not observed).
The Athene system had a high error rate for the entailment preserving transformations: even though the model had an evidence recall of for the adversarial instances, the FEVER score reduced by . On inspection, this model is mostly predicting Supported for our adversarial instances. In contrast, the Papelo system had the lowest reduction () in FEVER score for entailment preserving adversarial instances, despite a reduction in evidence recall of . This model also exhibited a similar behaviour to the Athene model: while a large number of supported claims were correctly classified, this model predominantly predicted NotEnoughInfo for the adversarial instances. As this class does not require evidence to be correctly scored, ‘falling back’ to it resulted in a higher score for balanced data. This also explains the low FEVER score for complex negations (a reduction of ), in this evaluating this class of transformation, we only considered Supported and Refuted claims while the model was mostly predicting NotEnoughInfo.
6 Sample FEVER2.0 Shared Task Submission
We construct a balanced sample submission for the break-it phase of the FEVER2.0 shared task. We applied our rule-based transformations to the test set from  and then perform a stratified sample yielding 1000 instances. We compare this to a baseline that comprises a sample of the unmodified claims that reflects the current error rate of the systems prior to our adversarial attacks.
To verify the suitability of our method, we performed a manual evaluation of the claims generated by the rule based transformation. of the instances submitted was inspected and labelled as to whether the claims were grammatical. We found that (90%) of the claims met the submission guidelines and () of the claims did not. Of the claims that did not meet the guidelines due to being nonsensical or non-grammatical: was generated from an original claim that had a grammatical error; were made from claims with more complex language that was not accounted for when designing the rule-based transformations; and had a systematic error where the third-person neutral pronoun it was used for claims about people.
|1||Our Method (Rule-based Transformation)||62.58||90||56.32|
|2||Baseline (Unmodified Instances)||45.48||97||44.12|
Sample leader-board for the FEVER2.0 breakers. The adjusted potency score is an estimate based on the results of the manual inspection of the generated claims.
Considering the performance of the systems we test, we also present a leader-board ranked by the resilience (average FEVER score for all breaker methods). While the Papelo system was ranked on the FEVER shared task, this model performs the best in this evaluation. In Table 4, we presented a breakdown of results which indicated, that under entailment preserving transformations, this model performed better for the other systems we tested against. Entailment-preserving transformations accounted for around of the claims sampled our submission666Only the entailment preserving rules generated claims of all three classes Supported, Refuted and NotEnoughInfo whereas the negating transformations were only applied to the Supported and Refuted claims which may account for this result. The UCL system (ranked in the shared task) also performs better than UNC system (ranked in the shared task) in both this sample evaluation and the breakdown in Table 4. This may arise from the fact that both teams make use of the development data that we used to generate our claims differently (i.e. the pretrained UCL model we used incorporated development data to increase the size of the training set whereas other models did not). As it is not possible to control how this data is used in the FEVER2.0 shared task, our method for generating claims from transforming existing instances may perform better (i.e. have more potency) when applied to instances that have not previously been released to the public.
In this report, we explored the behaviour of systems for Fact Extraction and VERification under adversarial attacks by applying simple rule-based perturbations to existing instances. We showed that adversarial examples can be generated through very simple transformations that are low cost and generate grammatically correct instances. We hope that the findings that we present in this paper help continue the discussion on improving models to be robust and resilient to data from outside of the training distribution. The FEVER2.0 shared task is just one forum for discussing the robustness of models against adversarial examples. We hope that the techniques in this paper help inspire participation in the task and that the data generated by the shared task may be used by researchers in other disciplines to help characterize the data used to train their models and the potential risks of attack that these models may face when deployed in the wild.
The authors wish to thank Trevor Cohn, Tim Baldwin and the FEVER Organizers for their helpful advice.
- Belinkov and Bisk  Yonatan Belinkov and Yonatan Bisk. Synthetic and Natural Noise Both Break Neural Machine Translation. In ICLR, pages 1–13, 2018. URL http://arxiv.org/abs/1711.02173.
- Bowman et al.  Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. 2015. ISSN 9781941643327. URL http://arxiv.org/abs/1508.05326.
- Burlot and Yvon  Franck Burlot and François Yvon. Evaluating the morphological competence of Machine Translation Systems. In Proceedings of the Conference on Machine Translation (WMT), volume 1, pages 43–55, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/w17-4705.
- Chen et al.  Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1657–1668, Vancouver, Canada, 2016. ISBN 9781945626753. doi: 10.18653/v1/P17-1152. URL http://arxiv.org/abs/1609.06038.
- Ebrahimi et al.  Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-Box Adversarial Examples for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 31–36, Melbourne, Australia, 2018. Association for Computational Linguistics. URL http://arxiv.org/abs/1712.06751.
- Ettinger et al.  Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. Towards linguistically generalizable nlp systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1–10, 2017.
- Gardner et al.  Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. AllenNLP: A Deep Semantic Natural Language Processing Platform. 2017.
- Goodfellow et al.  Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412.6572.
- Gururangan et al.  Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation Artifacts in Natural Language Inference Data. 2018. ISSN 1702.00887. doi: 10.18653/v1/N18-2017. URL http://arxiv.org/abs/1803.02324.
- Hanselowski et al.  Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. Multi-sentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 103–108, Brussels, Belgium, November 2018. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/W18-5516.
- Isabelle et al.  Pierre Isabelle, Colin Cherry, and George Foster. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496, 2017.
- Iyyer et al.  Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1875–1885, 2018.
- Jia and Liang  Robin Jia and Percy Liang. Adversarial Examples for Evaluating Reading Comprehension Systems. 2017. URL http://arxiv.org/abs/1707.07328.
- Levesque  Hector J Levesque. On our best behaviour. IJCAI, 2013. ISSN 0035-8797.
- Mahler et al.  Taylor Mahler, Willy Cheung, Micha Elsner, David King, Marie-Catherine de Marneffe, Cory Shain, Symon Stevens-Guille, and Michael White. Breaking nlp: Using morphosyntax, semantics, pragmatics and world knowledge to fool sentiment analysis systems. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 33–39, 2017.
Team papelo: Transformer networks at fever.In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 109–113, Brussels, Belgium, November 2018. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/W18-5517.
- Miyato et al.  Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. Adversarial training methods for semi-supervised text classification. ICLR, 2017. URL https://arxiv.org/abs/1605.07725.
- Naik et al.  Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress Test Evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, 2018. URL http://arxiv.org/abs/1806.00692.
Nie et al. 
Yixin Nie, Haonan Chen, and Mohit Bansal.
Combining fact extraction and verification with neural semantic
Association for the Advancement of Artificial Intelligence (AAAI), 2019.
- Parikh et al.  Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A Decomposable Attention Model for Natural Language Inference. pages 2249–2255, 2016. ISSN 0001-0782. doi: 10.18653/v1/N16-1062. URL http://arxiv.org/abs/1606.01933.
- Peters et al.  Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. 2018. ISSN 9781941643327. doi: 10.18653/v1/N18-1202. URL http://arxiv.org/abs/1802.05365.
- Poliak et al.  Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis Only Baselines in Natural Language Inference. (1):180–191, 2018. ISSN 0885-3924. URL http://arxiv.org/abs/1805.01042.
- Rajpurkar et al.  Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. pages 2383–2392, 2016. URL http://arxiv.org/abs/1606.05250.
- Ribeiro et al.  Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically Equivalent Adversarial Rules for Debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/P18-1079.
- Staliūnaite and Bonfil  Ieva Staliūnaite and Ben Bonfil. Breaking sentiment analysis of movie reviews. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 61–64, 2017.
- Szegedy et al.  Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http://arxiv.org/abs/1312.6199.
- Thorne and Vlachos  James Thorne and Andreas Vlachos. Automated Fact Checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346–3359, Santa Fe, New Mexico, USA, 2018. Association for Computational Linguistics. URL http://aclweb.org/anthology/C18-1283.
- Thorne et al. [2018a] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana, 2018a. Association for Computational Linguistics. URL http://aclweb.org/anthology/N18-1074.
- Thorne et al. [2018b] James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. The Fact Extraction and VERification (FEVER) Shared Task. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 1–9, Brussels, Belgium, 2018b. Association for Computational Linguistics. URL http://fever.ai/.
- Williams et al.  Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.
- Yoneda et al.  Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Ucl machine reading group: Four factor framework for fact finding (hexaf). In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 97–102, Brussels, Belgium, November 2018. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/W18-5515.
- Zellers et al.  Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/D18-1009.
- Zhao et al.  Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating Natural Adversarial Examples. ICLR, 2018. ISSN 0932-433X. doi: 10.1007/s00482-008-0658-4. URL http://arxiv.org/abs/1710.11342.