Imagine a woman chasing a dog around outside, trying to give it a bath. What might happen next? Humans can read a narrative like this, shown in Figure 1, and connect it to a rich model of the world: the dog is currently dry and not soapy, and it actively doesn’t want to be bathed. Thus, one plausible next event is option C—that she’ll get the dog wet and it will run away again.
When the SWAG dataset was first announced Zellers et al. (2018), this new task of commonsense natural language inference seemed trivial for humans (88%) and yet challenging for then-state-of-the-art models (60%), including ELMo Peters et al. (2018). However, BERT Devlin et al. (2018) soon reached over 86%, almost human-level performance. One news article on this development was headlined “finally, a machine that can finish your sentence.”111A New York Times article at https://nyti.ms/2DycutY.
In this paper, we investigate the following question: How well do deep pretrained models, like BERT, perform at commonsense natural language inference (NLI)? Our surprising conclusion is that the underlying task remains unsolved. Indeed, we find that deep models such as BERT do not demonstrate robust commonsense reasonining ability by themselves. Instead, they operate more like rapid surface learners for a particular dataset. Their strong performance on SWAG is dependent on the finetuning process, wherein they largely learn to pick up on dataset-specific distributional biases. When the distribution of language shifts slightly, performance drops drastically – even if the domain remains identical.
We study this question by introducing HellaSwag,222Short for Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations. Dataset and code at https://rowanzellers.com/hellaswag. a new benchmark for commonsense NLI. We use Adversarial Filtering (AF), a data-collection paradigm in which a series of discriminators is used to select a challenging set of generated wrong answers. AF is surprisingly effective towards this goal: the resulting dataset of 70k problems is easy for humans (95.6% accuracy), yet challenging for machines (. This result holds even when models are given a significant number of training examples, and even when the test data comes from the exact same distribution as the training data. Machine performance slips an additional 5% when evaluated on examples that cover novel concepts from the same domain.
To make this dataset robust to deep pretrained models, we use a trifecta of state-of-the-art generators Radford et al. (2018), state-of-the-art discriminators (BERT), and high quality source text. We expand on the SWAG’s original video-captioning domain by using WikiHow articles, greatly increasing the context diversity and generation length. Our investigation reveals a Goldilocks zone – roughly three sentences of context, and two generated sentences – wherein generations are largely nonsensical, even though state-of-the-art discriminators cannot reliably tell the difference between these generations and the ground truth.
More broadly, our paper presents a case-study towards a future of verified progress in NLP, via iterative rounds of building and breaking datasets. If our ultimate goal is to provide reliable benchmarks for challenging tasks, such as commonsense NLI, these benchmarks cannot be static. Instead, they must evolve together with the evolving state-of-the-art. Continued evolution in turn requires principled dataset creation algorithms. Whenever a new iteration of a dataset is created, these algorithms must leverage existing modeling advancements to filter out spurious biases. Only once this cycle becomes impossible can we say that the underlying task – as opposed an individual dataset – is solved.
SWAG is a dataset for commonsense NLI. For each question, a model is given a context from a video caption and four ending choices for what might happen next. Only one choice is right – the actual next caption of the video.
Obtaining interesting negatives is challenging. Prior work (e.g. Gururangan et al., 2018; Poliak et al., 2018) has found that when humans write the endings to NLI questions, they introduce subtle yet strong class-conditional biases known as annotation artifacts.333These biases simply inflate model performance, but past work has also shown that are unwanted social biases induced when humans write the endings, in terms of gender and race Rudinger et al. (2015).
To address this, Zellers et al. (2018) introduced Adversarial Filtering (AF). An overview is shown in Figure 2. The key idea is to produce a dataset which is adversarial for any arbitrary split of , . This requires a generator of negative candidates (i.e., wrong endings that violate human notions about how the world works), which we achieve by using a language model. Potential candidates of incorrect answers were massively oversampled from a language model trained on in-domain data, and then selected using an ensemble of adversaries. The selection process happens iteratively: on each iteration, the dataset is randomly partitioned into and . The ensemble is trained to classify endings as real or generated on , then, AF replaces easy-to-classify generations in . This process continues until the accuracy of these adversaries converges. Last, humans validate the data to remove adversarial endings that seem realistic.
Importantly, AF creates a final dataset that is challenging to models regardless of the final dataset split. In Section 4, we will use AF as the underlying workhorse to construct an NLI dataset that is easy for humans, yet challenging for machines. This difficulty persists even when models are provided significant training data, and even when this data comes from the same distribution as the test set. This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.
3 Investigating SWAG
In this section, we investigate why SWAG was solved. We focus on BERT, since it is the best known approach at the time of writing.444 See the appendix for a discussion of the BERT architecture and hyperparameter settings we used in our experiments.
See the appendix for a discussion of the BERT architecture and hyperparameter settings we used in our experiments.Core to our analysis is investigating how a model trained on Wikipedia and books can be so effectively finetuned for SWAG, a dataset from video captions.
3.1 How much innate knowledge does BERT have about SWAG?
We investigate this question by measuring BERT’s performance on SWAG while varying the size of the training dataset; results are shown in Figure 3. While the best known ELMo NLI model (ESIM+ELMo; Chen et al., 2017) requires the entire training set to reach 59%, BERT outperforms this given only 64 examples. However, BERT still needs upwards of 16k examples to approach human performance, around which it plateaus.
3.2 What is learned during finetuning?
|Ending Only||No context is provided; just the endings.|
|Shuffled||Endings that are indidivually tokenized, shuffled, and then detokenized.|
|Shuffled+ Ending Only||No context is provided and each ending is shuffled.|
Figure 4 compares BERT’s performance when trained and evaluated on variants of SWAG.
Context: BERT’s performance only slips 11.9 points (86.7%74.8%) when context is omitted (Ending Only), suggesting a bias exists in the endings themselves.555These biases are similar to those in NLI datasets, as found by Gururangan et al. (2018); Poliak et al. (2018). If a followup event seems unreasonable absent of context, then there must be something markedly different between the space of human-written and machine-generated endings.
Structure: To distinguish word usage from structural patterns, we consider a new scenario, Shuffled. Here the shared context is provided, but the words in each ending choice are randomly permuted. Surprisingly, this reduces BERT performance by less than 10%. Even though BERT was never exposed to randomly shuffled text during pretraining, it easily adapts to this setting, which suggests that BERT is largely performing lexical reasoning over each (context, answer) pair.
Finally, when the context is removed and the words in each ending are shuffled, performance drops to 60.4%. While low, this is still higher than ELMo’s performance (% from Zellers et al., 2018). As neither context nor structure is needed to discriminate between human and machine-written endings in a majority of cases, it is likely that systems primarily learn to detect distributional stylistic patterns during finetuning.
3.3 Where do the stylistic biases come from?
SWAG was constructed via Adversarial Filtering (AF). Endings were generated via a language model, and then selected to fool a discriminator. To understand why it was solved requires understanding the interplay of AF with respect to SWAG’s generators and discriminators.
Zellers et al. (2018) used a two-layer LSTM for generation, with shallow stylistic adversarial filters.666 The discriminator was an ensemble that featured a bag of words model, a shallow CNN, a multilayer perceptron operating on language model perplexities.
The discriminator was an ensemble that featured a bag of words model, a shallow CNN, a multilayer perceptron operating on language model perplexities.This setup was robust against ELMo models, but has the shallow LM in particular produced distributional artifacts that BERT picks up on?
To investigate this, we perform AF using BERT-Large as the discriminator777On each iteration, BERT-Large is re-initialized from its pretrained checkpoint, finetuned, and then evaluated in a four-way setting on the dummy test set of held-out data. See Supp A for a details of our BERT-Large AF setup. in two settings, comparing generations from Zellers et al. (2018) with those from a finetuned GPT Radford et al. (2018).
Strikingly, the results, Figure 5 (left), show that the generations used in SWAG are so different from the human-written endings that AF never drops the accuracy to chance; instead, it converges to roughly 75%. On the other hand, GPT’s generations are good enough that BERT accuracy drops below 30% over many random subsplits of the data, revealing the importance of the generator.
The success of BERT implies that high-quality generators and discriminators are crucial to AF’s success. However, it does not imply that the underlying task of commonsense NLI – as opposed to a single dataset – is solved. To evaluate this claim requires us to try making a new evolution of the SWAG dataset, one in which artifacts are removed. In this section, we do just that by introducing HellaSwag.
4.1 ActivityNet Captions
We start by including video captions from the ActivityNet Captions dataset Krishna et al. (2017). The original SWAG dataset contains these, along with captions from LSMDC Rohrbach et al. (2017), but for HellaSwag we solely used ActivityNet. In addition to temporal descriptions, ActivityNet also provides activity labels for each caption (e.g. jumping rope). We will use these activity labels as additional structure to test generalization ability.
4.2 WikiHow: A New Testbed
We next consider a new and challenging testbed for commonsense reasoning: completing how-to articles from WikiHow, an online how-to manual. We scrape 80k context and follow-up paragraphs from WikiHow, covering such diverse topics as “how to make an origami owl” to “how to survive a bank robbery.” Each context has at most three sentences, as do the follow-ups.
AF’s effectiveness in this new setting is shown in Figure 5 (right). We consider three settings, corresponding to endings that are either one, two, or three sentences long. In all cases, BERT performance begins high (70-90%), but there are enough generations for Adversarial Filtering to lower the final accuracy considerably. While the one-sentence case converges to slightly higher than random – 35% when it converges – the two and three sentence cases are higher, at 40% and 50% respectively. Given more context, it becomes easier to classify an ending as machine- or human-written. We compromise and use two-sentence generations. Particularly in the two-sentence case, we find ourselves in a Goldilocks zone wherein generations are challenging for deep models, yet as we shall soon see, easy for humans.
4.3 Obtaining high human agreement
How well can humans distinguish human-written endings from machine generations refined with Adversarial Filtering? In Figure 6, we compare human performance with that of BERT on a random 80%/20% split. We see a contrast between the ActivityNet and WikiHow performance. While ActivityNet starts off harder for BERT (25.5%), it also proves difficult for humans (60%). In contrast, WikiHow starts easier for BERT (41.1%) and humans find the domain almost trivial (93.5%). We hypothesis this discrepancy is due to the lengths of both datasets (Figure 7). WikiHow’s 2-sentence generations average 41 tokens, versus 13 for ActivityNet. This gives WikiHow generations three times as many opportunities to make a detectable mistake.
To ensure high agreement on ActivityNet, we perform several rounds of human filtering, increasing human performance to 94%. During human validation, crowd workers are given a context and six ending choices, of which one is the true ending, and the other five are from AF. On each iteration, we replace machine-written endings that the worker rated as realistic with new samples. In the end, we keep the 25k best ActivityNet contexts (i.e. those with highest agreement among workers
888 See the appendix for details about how we estimate this.
See the appendix for details about how we estimate this.) and the 45k best WikiHow contexts.
4.4 Zero-shot categories for evaluation
To evaluate a model’s ability to generalize to new situations, we use category labels from WikiHow and ActivityNet to make ‘zero-shot’ evaluation sets. For each set (validation or test), we craft two subsets: one containing 5k ‘in-domain’ examples that come from categories as seen during training (Figure 8), and another with 5k ‘zero-shot’ examples from randomly chosen held-out categories. In total, there are 70k dataset examples.
We evaluate the difficulty of HellaSwag using a variety of strong baselines, with and without massive pretraining. The models share the same format: given a context and an ending, return a logit for that ending. Accordingly, we train our models using a four-way cross-entropy loss, where the objective is to predict the correct ending. In addition to BERT-Large, our comparisons include:
Bert-Base: A smaller version of the BERT model whose architecture size matches GPT.
LSTM sentence encoder
: This is a randomly initialized two-layer bi-LSTM; the second layer’s hidden states are max-pooled and fed into an MLP to predict the logit. We consider three variations: GloVe embeddings, ELMo embeddings, or (frozen) BERT-Base embeddings.999For ELMo and BERT-Base, the model learns scalar weights to combine each internal layer of the encoder.
FastText: Joulin et al. (2017) An off-the-shelf library for bag-of-words text classification.101010This model is trained with binary cross entropy loss.
We compare all models to human performance by asking five independent crowd workers to solve the same four-way multiple choice problems; their predictions are combined via majority vote.
Our results, shown in Table 1, hint at the difficulty of the dataset: human performance is over 95%, while overall model performance is below 50% for every model. Surprisingly, despite BERT-Large having been used as the adversarial filter, it still performs the strongest at 47.3% overall. By making the dataset adversarial for BERT, it seems to also have become adversarial for every other model. For instance, while ESIM+ELMo obtained 59% accuracy on SWAG, it obtains only 33.3% accuracy on HellaSwag.
In addition to pretraining being critical, so too is end-to-end finetuning. Freezing BERT-Base and adding an LSTM on top lowers its overall performance 4.3%. This may help explain why models such as ESIM+ELMo struggled on SWAG, as ELMo isn’t updated during finetuning.
While BERT is the best model, it still struggles on HellaSwag, and especially so on zero-shot categories. Performance drops roughly 5% on the test fold, which suggests that the finetuning is not enough for BERT to learn to generalize to novel activities or how-to categories.
Last, we see that WikiHow is a much harder domain that ActivityNet for machines: 45% Bert-Large performance, versus 96.5% for humans. Curiously, it is on this source dataset that we see the smallest gap between OpenAI GPT and BERT. In fact, OpenAI GPT outperforms BERT on WikiHow, but the reverse is true for ActivityNet. One possibility is that the left-to-right structure of GPT is the right inductive bias for WikiHow - perhaps reasoning bidirectionally over long contexts is too much for a 12-layer transformer to learn.
5.1 SWAG to HellaSwag transfer
Given the shared goals and partial domains of SWAG and HellaSwag, it is natural to ask to what extent models can transfer between the two datasets.
In Figure 9 we show the results from transfer experiments: models are trained on one dataset and evaluated on the other.111111 Note that the ActivityNet splits are different for each dataset. To avoid skewing the results, we report only on the validation video captions that are not in the training sets of either dataset. The overall accuracy is then a weighted average, where ActivityNet examples are weighted proportionately more. This gives a slight advantage to training on SWAG, as it sees all the ActivityNet categories when training.
Note that the ActivityNet splits are different for each dataset. To avoid skewing the results, we report only on the validation video captions that are not in the training sets of either dataset. The overall accuracy is then a weighted average, where ActivityNet examples are weighted proportionately more. This gives a slight advantage to training on SWAG, as it sees all the ActivityNet categories when training.
The best models are trained on the same dataset that they are evaluated on: training on SWAG and evaluating on HellaSwag lowers performance by 12%; vice versa lowers performance by 15%. The missing domain for HellaSwag models is movie descriptions (LSMDC), still, HellaSwag models obtain 69% accuracy. On the other hand, SWAG models do not generalize at all to their missing domain, WikiHow (28%), suggesting that learning general commonsense reasoning was hardly necessary to solve SWAG.
5.2 Qualitative examples
We show several qualitative examples in Figure 10, along with BERT-Large’s predictions. BERT does well on some ActivityNet contexts, such as in the first row, where it correctly predicts the ending for a shaving caption. Whereas shaving is in-domain, the second example about sharpening knives is zero-shot. In this context, BERT’s answer suggests that one would use a knife to sharpen a stone, rather than vice versa. The last example comes from WikiHow, which appears to be incredibly challenging for BERT. BERT picks answer d, which has more words that match the context of technology (planes, traffic, laptop), but is incoherent.121212Among other issues, why would someone suddenly be aware that they are ‘flying at high speed on a plane…?’
Our results suggest that HellaSwag is a challenging testbed for state-of-the-art NLI models, even those built on extensive pretraining. The question still remains, though, of where will the field go next?
6.1 How easy might HellaSwag be for future discriminators?
In this paper, we showed the existence of a Goldilocks zone of text complexity – in which generations are nonsensical, but existing state-of-the-art NLP models cannot tell the difference. How hard will the dataset be for future, even more powerful, models?
Answering this question is challenging because these models don’t exist (or are unavailable) at the time of writing. However, one remedy is to perform an ablation study on the Adversarial Filtering model used, comparing weaker filters with stronger discriminators. We present our results in Figure 11, and find that while weak discriminators (like the stylistic ensemble used to make SWAG) only marginally reduce the accuracy of BERT-Large, increasing the gap between the filter and the final discriminator is not enough to solve the task. For instance, using a discriminator with 3x the parameters as the adversarial filter (BERT-Large vs. BERT-Base) results in 63% machine accuracy.
6.2 How well does pretraining scale?
Overall, the current paradigm of pretraining large models on lots of data has made immense progress on NLP benchmarks. Though we expect this trend to continue, it also behooves us to consider its limits. If more compute is indeed the answer for human-level commonsense inference, what would the compute requirements of this hypothetical massive model look like?
We investigate this in Figure 12 by comparing the accuracies of known models on HellaSwag with their computational needs. This estimation is a rough estimate: we convert reported TPU runtimes to our benchmark RTX 2080 Ti GPU using the Roofline model Williams et al. (2009)
, which focuses primarily on the bottleneck of loading tensors into GPU memory. Extrapolating from an exponential fit suggests that reaching human-level performance on our dataset would requireGPU hours, or 100k years – unless algorithmic improvements are made.
What might these algorithmic improvements look like? These could include architectural advances, better pretraining objectives, and beyond. However, these improvements share the bottleneck of the data source. To answer some HellaSwag questions correctly without reasoning deeply – like knowing that it is a bad idea to stop at a red light for ‘at most two seconds’ – might require an exponential number of samples, due to problems of reporting bias Gordon and Van Durme (2013). Alternatively, future models might answer correctly only by picking up on spurious patterns, in which case a new development of the benchmark – using these models as adversaries – would place us in the same position as we are right now.
Put another way, for humans to answer HellaSwag questions requires abstracting away from language and modeling world states instead. We postulate that this is what separates solving the task of commonsense NLI, as opposed to a particular dataset. Indeed, we find that existing deep methods often get fooled by lexical false friends. For example, in the WikiHow example from Figure 10, BERT chooses an ending that matches the technology words in the context, rather than matching the deeper topic: using technology as an excuse for not doing homework.
6.3 Towards a future of evolving benchmarks
What happens when HellaSwag gets solved? We believe the answer is simple: crowdsource another dataset, with the same exact format, and see where models fail. Indeed, in our work we found this to be straightforward from an algorithmic perspective: by throwing in the best known generator (GPT) and the best known discriminator (BERT-Large), we made a dataset that is adversarial - not just to BERT, but to all models we have access to.
While this was easy algorithmically, care must be taken from a data curation standpoint. Indeed, we find success exists within a Goldilocks zone: the data source must be complex enough that state-of-the-art generators often make mistakes, while simple enough such that discriminators often fail to catch them. This ties the future of SWAG-style benchmarks to progress on language generation: until generation is solved, commonsense NLI will remain unsolved. Even recent promising results on scaling up language models Radford et al. (2019) find problems in terms of consistency, with the best curated examples requiring 25 random seeds.
In this paper, we presented HellaSwag, a new dataset for physically situated commonsense reasoning. By constructing the dataset through adversarial filtering, combined with state-of-the-art models for language generation and discrimination, we produced a dataset that is adversarial to the most robust models available – even when models are evaluated on items from the training distribution. In turn, we provided insight into the inner workings of pretrained models, and suggest a path for NLP progress going forward: towards benchmarks that adversarially co-evolve with evolving state-of-the-art models.
We thank the reviewers, as well as Jesse Thomason, for their helpful feedback. We thank the Mechanical Turk workers for their great work during dataset collection. Thanks also to Zak Stone and the Google Cloud TPU team for help with the computing infrastructure. This work was supported by the National Science Foundation through a Graduate Research Fellowship (DGE-1256082) and NSF grants (IIS-1524371, 1637479, 165205, 1703166), the DARPA CwC program through ARO (W911NF-15-1-0543), the IARPA DIVA program through D17PC00343, the Sloan Research Foundation through a Sloan Fellowship, the Allen Institute for Artificial Intelligence, the NVIDIA Artificial Intelligence Lab, and gifts by Google and Facebook. The views and conclusions contained herein are those of the authors and should not be interpreted as representing endorsements of IARPA, DOI/IBC, or the U.S. Government.
Belinkov and Bisk (2018)
Yonatan Belinkov and Yonatan Bisk. 2018.
Synthetic and natural noise both break neural machine translation.In ICLR. ICLR.
- Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1657–1668.
- Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655.
- Gordon and Van Durme (2013) Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25–30. ACM.
- Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proc. of NAACL.
- Holtzman et al. (2019) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Jia and Liang (2017)
Robin Jia and Percy Liang. 2017.
Adversarial examples for evaluating reading comprehension systems.
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031.
- Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427–431.
Krishna et al. (2017)
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.
Dense-Captioning Events in Videos.
International Conference on Computer Vision (ICCV).
- Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227–2237.
- Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191.
- Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI.
- Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
- Rohrbach et al. (2017) Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie Description. International Journal of Computer Vision, 123(1):94–120.
- Rudinger et al. (2015) Rachel Rudinger, Vera Demberg, Ashutosh Modi, Benjamin Van Durme, and Manfred Pinkal. 2015. Learning to predict script events from domain-specific text. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 205–210.
- Williams et al. (2009) Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: An insightful visual performance model for floating-point programs and multicore architectures. Technical report, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States).
- Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724.
Appendix A Adversarial Filtering Setup
In this subsection, we provide some more details regarding the Adversarial Filtering experiments.
Our version of Adversarial Filtering is mostly the same as Zellers et al. (2018). Details:
[wide, labelwidth=!,labelindent=0pt,noitemsep,topsep=0pt,leftmargin =*,label=.]
On each iteration, we split the dataset up into 80% training and 20% testing. We don’t do anything special for this split (like looking at the video/article IDs).
For ActivityNet, we use assigned indices for every example. (This corresponds to the number of red columns in Figure 2). For WikiHow, we used , since we found that there were fewer good endings produced by the generators after scaling up the sequence length.
’s model for SWAG: on each training example, the model is given exactly one positive ending and several negative endings, and the model computes probability distribution over the endings through a softmax. However, we also wanted to always report 4-way probability for simplicity. To do this, we train in a 4-way setting (the training set is constructed by subsamplingwrong answers from the set of that are currently assigned to each example). The accuracy values that are reported are done so using the first 3 assigned negatives in dataset .
Sometimes, BERT never converges (accuracy around 25%), so when this happens, we don’t do the reassignment.
Appendix B GPT Setup
We generate our dataset examples from OpenAI GPT. We finetune the model for two epochs on WikiHow, and 5 epochs on ActivityNet, using the default learning rate ofRadford et al. (2018). Importantly, we generate randomly according to the language model distribution, rather than performing beam search – this would bias the generations towards common words. For the WikiHow endings, we used Nucleus Sampling with , which means that the probability weights for the tail (those tokens with cumulative probability mass ) are zeroed out (Holtzman et al., 2019).
Appendix C BERT setup
We extensively study BERT in this paper, and make no changes to the underlying architecture or pretraining. For all of the experiments where we provide context, we set up the input to the BERT model like this:
[CLS] A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. [SEP] She gets the dog wet, then it runs away again [SEP]
In the case where only the ending is provided, we adopt the BERT-style ‘single-span’ setting: [CLS] She gets the dog wet, then it runs away again [SEP]
Appendix D A discussion on BERT Hyperparameters and Instability
It is worth noting that many of our experiments some instability. On the SWAG experiments, we use the same hyperparameters as Devlin et al. (2018) - these generally work very well.131313The only exception is for the plots where we vary the number of training examples. In this case, we don’t want to disadvantage the trials without much training data (since this would allow for fewer parameter updates). To remedy this, we continue training for 10 epochs and report the best validation performance over the entire training history. However, we find that they become a bit unstable when crossing over to make HellaSwag. Here, we discuss some strategies and insight that we picked up on.
[wide, labelwidth=!,labelindent=0pt,noitemsep,topsep=0pt,leftmargin =*,label=.]
We use a batch size of 64 examples rather than 16, and warm the model up for 20% of the dataset (rather than 10%). This helps the model adapt to SWAG more gradually, without diverging early on.
For the Adversarial Filtering experiments (for both WikiHow and ActivityNet), we randomize some of the hyperaparmeters on each iteration. We sample a learning rate between 1e-5 and 4e-5
, using a log-uniform distribution. These outer ranges were recommended from the original BERT paper. Additionally, with probabilitywe use the cased model (where the input isn’t originally lowercased before tokenization), rather than the uncased model.
During adversarial filtering, we used 3 epochs. However, we found that adding more epochs helped the model during fine-tuning on the final dataset HellaSwag. Our best configuration uses 10 epochs.
While fine-tuning on HellaSwag we used a learning rate of 2e-5.
Appendix E Human validation
We performed human validation using the same setup as Zellers et al. (2018). Humans get six answers to choose from, of which exactly one is the true ending and the other five are from AF. We found that multiple rounds of human validation were especially helpful on ActivityNet. However, it helps to do the human validation in an intelligent way: if the first worker is confused, the answer should be replaced before it goes to the next worker. This is a hard problem, so we adopt the following approach:
[wide, labelwidth=!,labelindent=0pt,noitemsep,topsep=0pt,leftmargin =*,label=.]
We use best practices on mechanical turk, paying workers fairly (up to 37 cents per HIT on WikiHow). We also used a qualification HIT that was autograded to help filter for workers who are good at the task. Workers who tended to prefer the generated endings over the real ones were dequalified from participating.
For each worker, we use the summary of their performance so far to estimate . We can then use this to estimate how confident we are in each answer choice: we want to be confident that workers will not prefer the wrong answers. Also, this allows us to aggregate performance across crowd workers, by multiplying the probabilities for each answer choice.
On each round of filtering, we keep the 3 wrong endings that workers least prefer (based on the probability scores, along with the right ending. The other two endings are new ones.
Particularly on ActivityNet, we found that there are some contexts where the ground truth answer isn’t liked by workers. To fix this, we end up taking the best 25k examples from ActivityNet and the best 45k from WikiHow. (By best, we mean the ones with the highest probability that workers will predict the true answer, versus the three easiest-to-guess negatives, as judged by the Naive Bayes model). We make Figure 7 (‘The road to HellaSwag’) by doing this process (taking the best examples) for each dataset, while varying the number of annotators that are used for getting the scores for each ending. (In the case where there are 0 annotators, we get a random sample).
Appendix F Human Evaluation
We do a human evaluation while giving workers the exact same task as is given to the models. Workers are given five endings, and must pick the best one. We obtain human evaluation numbers by combining 5 turkers together, with a majority vote.
We found that the biggest differences in difficulty in humans were due to domain (WikiHow is easier than ActivityNet). To account for this, we did the human evaluation over 200 examples from WikiHow, and 200 examples from ActivityNet, for each number of previous validators as shown in Figure 7 (0, 1, or 2). To report the accuracy of a split that’s mixed between WikiHow and ActivityNet, we use the following formula:
Here, refers to the accuracy on each dataset as judged by humans, and is the number of examples from that dataset in the split.
Appendix G More examples
We additionally have more validation examples, shown in Figure 2.
Appendix H In-Domain and Zero-Shot categories
See Figure 13 for a closer look at the dataset categories.