Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

09/16/2021
by   Hendrik Schuff, et al.
0

Natural language inference (NLI) requires models to learn and apply commonsense knowledge. These reasoning abilities are particularly important for explainable NLI systems that generate a natural language explanation in addition to their label prediction. The integration of external knowledge has been shown to improve NLI systems, here we investigate whether it can also improve their explanation capabilities. For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess fine-grained reasoning capabilities. We find that different sources of knowledge have a different effect on reasoning abilities, for example, implicit knowledge stored in language models can hinder reasoning on numbers and negations. Finally, we conduct the largest and most fine-grained explainable NLI crowdsourcing study to date. It reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/10/2021

How Commonsense Knowledge Helps with Natural Language Tasks: A Survey of Recent Resources and Methodologies

In this paper, we give an overview of commonsense reasoning in natural l...
research
07/23/2023

CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Models

Recently, large pretrained language models have achieved compelling perf...
research
05/09/2023

MoT: Pre-thinking and Recalling Enable ChatGPT to Self-Improve with Memory-of-Thoughts

Large Language Models have shown impressive abilities on various tasks. ...
research
12/12/2022

Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments

The spread of misinformation, propaganda, and flawed argumentation has b...
research
07/15/2021

Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference Task

The recent state-of-the-art natural language understanding (NLU) systems...
research
10/12/2020

Social Commonsense Reasoning with Multi-Head Knowledge Attention

Social Commonsense Reasoning requires understanding of text, knowledge a...
research
10/24/2020

ANLIzing the Adversarial Natural Language Inference Dataset

We perform an in-depth error analysis of Adversarial NLI (ANLI), a recen...

Please sign up or login with your details

Forgot password? Click here to reset