Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED

04/17/2022
by   Quzhe Huang, et al.
0

DocRED is a widely used dataset for document-level relation extraction. In the large-scale annotation, a recommend-revise scheme is adopted to reduce the workload. Within this scheme, annotators are provided with candidate relation instances from distant supervision, and they then manually supplement and remove relational facts based on the recommendations. However, when comparing DocRED with a subset relabeled from scratch, we find that this scheme results in a considerable amount of false negative samples and an obvious bias towards popular entities and relations. Furthermore, we observe that the models trained on DocRED have low recall on our relabeled dataset and inherit the same bias in the training data. Through the analysis of annotators' behaviors, we figure out the underlying reason for the problems above: the scheme actually discourages annotators from supplementing adequate instances in the revision phase. We appeal to future research to take into consideration the issues with the recommend-revise scheme when designing new models and annotation schemes. The relabeled dataset is released at <https://github.com/AndrewZhe/Revisit-DocRED>, to serve as a more reliable test set of document RE models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/25/2022

Revisiting DocRED – Addressing the Overlooked False Negative Problem in Relation Extraction

The DocRED dataset is one of the most popular and widely used benchmarks...
research
06/16/2023

Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data

Relation extraction (RE) aims to extract relations from sentences and do...
research
12/07/2020

H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extraction

Although distant supervision automatically generates training data for r...
research
04/17/2021

DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extraction

Distant supervision (DS) is a well established technique for creating la...
research
10/16/2019

FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

We present FewRel 2.0, a more challenging task to investigate two aspect...
research
10/28/2022

DORE: Document Ordered Relation Extraction based on Generative Framework

In recent years, there is a surge of generation-based information extrac...

Please sign up or login with your details

Forgot password? Click here to reset