Anatomy of OntoGUM–Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms

10/12/2021
by   YIlun Zhu, et al.
0

SOTA coreference resolution produces increasingly impressive scores on the OntoNotes benchmark. However lack of comparable data following the same scheme for more genres makes it difficult to evaluate generalizability to open domain data. Zhu et al. (2021) introduced the creation of the OntoGUM corpus for evaluating geralizability of the latest neural LM-based end-to-end systems. This paper covers details of the mapping process which is a set of deterministic rules applied to the rich syntactic and discourse annotations manually annotated in the GUM corpus. Out-of-domain evaluation across 12 genres shows nearly 15-20 systems, indicating a lack of generalizability or covert overfitting in existing coreference resolution models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/02/2021

OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres

SOTA coreference resolution produces increasingly impressive scores on t...
research
11/29/2022

End-to-End Neural Discourse Deixis Resolution in Dialogue

We adapt Lee et al.'s (2018) span-based entity coreference model to the ...
research
11/10/2021

A Novel Corpus of Discourse Structure in Humans and Computers

We present a novel corpus of 445 human- and computer-generated documents...
research
06/06/2017

Marmara Turkish Coreference Corpus and Coreference Resolution Baseline

We describe the Marmara Turkish Coreference Corpus, which is an annotati...
research
01/23/2019

Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge

This paper provides a detailed summary of the first shared task on End-t...
research
09/02/2019

Beyond The Wall Street Journal: Anchoring and Comparing Discourse Signals across Genres

Recent research on discourse relations has found that they are cued not ...
research
06/10/2021

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for b...

Please sign up or login with your details

Forgot password? Click here to reset