Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction

by   Tong Zhu, et al.

In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences. However, these DS-generated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers. To solve this problem, we build a new dataset NYTH, where we use the DS-generated data as training data and hire annotators to label test data. Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation. Finally, we present the experimental results of several widely used systems on NYT-H. The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different. This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction.



There are no comments yet.


page 1

page 2

page 3

page 4


Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

Distant supervision has been a widely used method for neural relation ex...

On the Effects of Low-Quality Training Data on Information Extraction from Clinical Reports

In the last five years there has been a flurry of work on information ex...

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction

Distantly supervised (DS) relation extraction (RE) has attracted much at...

Looking Beyond Label Noise: Shifted Label Distribution Matters in Distantly Supervised Relation Extraction

In recent years there is surge of interest in applying distant supervisi...

Re-TACRED: Addressing Shortcomings of the TACRED Dataset

TACRED is one of the largest and most widely used sentence-level relatio...

Indirect Supervision for Relation Extraction using Question-Answer Pairs

Automatic relation extraction (RE) for types of interest is of great imp...

InSRL: A Multi-view Learning Framework Fusing Multiple Information Sources for Distantly-supervised Relation Extraction

Distant supervision makes it possible to automatically label bags of sen...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.