Embracing Ambiguity: Shifting the Training Target of NLI Models

by   Johannes Mario Meissner, et al.

Natural Language Inference (NLI) datasets contain examples with highly ambiguous labels. While many research works do not pay much attention to this fact, several recent efforts have been made to acknowledge and embrace the existence of ambiguity, such as UNLI and ChaosNLI. In this paper, we explore the option of training directly on the estimated label distribution of the annotators in the NLI task, using a learning loss based on this ambiguity distribution instead of the gold-labels. We prepare AmbiNLI, a trial dataset obtained from readily available sources, and show it is possible to reduce ChaosNLI divergence scores when finetuning on this data, a promising first step towards learning how to capture linguistic ambiguity. Additionally, we show that training on the same amount of data but targeting the ambiguity distribution instead of gold-labels can result in models that achieve higher performance and learn better representations for downstream tasks.



There are no comments yet.


page 8


Addressing Ambiguity of Emotion Labels Through Meta-learning

Emotion labels in emotion recognition corpora are highly noisy and ambig...

Classification Under Ambiguity: When Is Average-K Better Than Top-K?

When many labels are possible, choosing a single one can lead to low pre...

Impossibility of Unambiguous Communication as a Source of Failure in AI Systems

Ambiguity is pervasive at multiple levels of linguistic analysis effecti...

Modeling Label Ambiguity for Neural List-Wise Learning to Rank

List-wise learning to rank methods are considered to be the state-of-the...

Deep Label Distribution Learning with Label Ambiguity

Convolutional Neural Networks (ConvNets) have achieved excellent recogni...

Multimodal Word Sense Disambiguation in Creative Practice

Language is ambiguous; many terms and expressions can convey the same id...

Sensing Ambiguity in Henry James' "The Turn of the Screw"

Fields such as the philosophy of language, continental philosophy, and l...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.