BenchIE: Open Information Extraction Evaluation Based on Facts, Not Tokens

by   Kiril Gashteovski, et al.

Intrinsic evaluations of OIE systems are carried out either manually – with human evaluators judging the correctness of extractions – or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of models' performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese and German. In contrast to existing OIE benchmarks, BenchIE takes into account informational equivalence of extractions: our gold standard consists of fact synsets, clusters in which we exhaustively list all surface forms of the same fact. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available.


page 1

page 2

page 3

page 4


AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark

Open Information Extraction (OIE) is the task of extracting facts from s...

Ditch the Gold Standard: Re-evaluating Conversational Question Answering

Conversational question answering (CQA) systems aim to provide natural-l...

X-FACT: A New Benchmark Dataset for Multilingual Fact Checking

In this work, we introduce X-FACT: the largest publicly available multil...

Towards Practical Evaluation of Android ICC Resolution Techniques

Inter-component communication (ICC) is a key mechanism in mobile apps, w...

ESBM: An Entity Summarization BenchMark

Entity summarization is the problem of computing an optimal compact summ...

On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?

Knowledge-grounded conversational models are known to suffer from produc...

Superpixels: An Evaluation of the State-of-the-Art

Superpixels group perceptually similar pixels to create visually meaning...