Embracing Ambiguity: Shifting the Training Target of NLI Models

06/06/2021
by   Johannes Mario Meissner, et al.
0

Natural Language Inference (NLI) datasets contain examples with highly ambiguous labels. While many research works do not pay much attention to this fact, several recent efforts have been made to acknowledge and embrace the existence of ambiguity, such as UNLI and ChaosNLI. In this paper, we explore the option of training directly on the estimated label distribution of the annotators in the NLI task, using a learning loss based on this ambiguity distribution instead of the gold-labels. We prepare AmbiNLI, a trial dataset obtained from readily available sources, and show it is possible to reduce ChaosNLI divergence scores when finetuning on this data, a promising first step towards learning how to capture linguistic ambiguity. Additionally, we show that training on the same amount of data but targeting the ambiguity distribution instead of gold-labels can result in models that achieve higher performance and learn better representations for downstream tasks.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

11/06/2019

Addressing Ambiguity of Emotion Labels Through Meta-learning

Emotion labels in emotion recognition corpora are highly noisy and ambig...
12/16/2021

Classification Under Ambiguity: When Is Average-K Better Than Top-K?

When many labels are possible, choosing a single one can lead to low pre...
08/02/2020

Impossibility of Unambiguous Communication as a Source of Failure in AI Systems

Ambiguity is pervasive at multiple levels of linguistic analysis effecti...
07/24/2017

Modeling Label Ambiguity for Neural List-Wise Learning to Rank

List-wise learning to rank methods are considered to be the state-of-the...
11/06/2016

Deep Label Distribution Learning with Label Ambiguity

Convolutional Neural Networks (ConvNets) have achieved excellent recogni...
07/15/2020

Multimodal Word Sense Disambiguation in Creative Practice

Language is ambiguous; many terms and expressions can convey the same id...
11/21/2020

Sensing Ambiguity in Henry James' "The Turn of the Screw"

Fields such as the philosophy of language, continental philosophy, and l...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.