Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

10/17/2020
by   Pengshuai Li, et al.
0

Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/14/2018

Improving Distantly Supervised Relation Extraction with Neural Noise Converter and Conditional Optimal Selector

Distant supervised relation extraction has been successfully applied to ...
research
10/30/2020

Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction

In recent years, distantly-supervised relation extraction has achieved a...
research
05/28/2020

Relation Extraction with Explanation

Recent neural models for relation extraction with distant supervision al...
research
09/05/2021

Semi-Automated Labeling of Requirement Datasets for Relation Extraction

Creating datasets manually by human annotators is a laborious task that ...
research
12/18/2020

Regularized Attentive Capsule Network for Overlapped Relation Extraction

Distantly supervised relation extraction has been widely applied in know...
research
08/21/2018

Neural Relation Extraction via Inner-Sentence Noise Reduction and Transfer Learning

Extracting relations is critical for knowledge base completion and const...
research
11/08/2022

Active Relation Discovery: Towards General and Label-aware Open Relation Extraction

Open Relation Extraction (OpenRE) aims to discover novel relations from ...

Please sign up or login with your details

Forgot password? Click here to reset