Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models

by   Viktor Schlegel, et al.

Recent years have seen a growing number of publications that analyse Natural Language Inference (NLI) datasets for superficial cues, whether they undermine the complexity of the tasks underlying those datasets and how they impact those models that are optimised and evaluated on this data. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets and the methods proposed to reveal and alleviate those weaknesses for the English language. We summarise and discuss the findings and conclude with a set of recommendations for possible future research directions. We hope it will be a useful resource for researchers who propose new datasets, to have a set of tools to assess the suitability and quality of their data to evaluate various phenomena of interest, as well as those who develop novel architectures, to further understand the implications of their improvements with respect to their model's acquired capabilities.



There are no comments yet.


page 1

page 2

page 3

page 4


Stress Test Evaluation for Natural Language Inference

Natural language inference (NLI) is the task of determining if a natural...

Continual Lifelong Learning in Natural Language Processing: A Survey

Continual learning (CL) aims to enable information systems to learn from...

Evolution of Semantic Similarity – A Survey

Estimating the semantic similarity between text data is one of the chall...

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

Integration of vision and language tasks has seen a significant growth i...

Neural Language Generation: Formulation, Methods, and Evaluation

Recent advances in neural network-based generative modeling have reignit...

A Survey of Current Datasets for Vision and Language Research

Integrating vision and language has long been a dream in work on artific...

Emerging Directions in Geophysical Inversion

In this chapter, we survey some recent developments in the field of geop...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.