Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction

05/23/2023
by   Ji Qi, et al.
0

The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a popular large language model, the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of 23.43 F1 score. Our resources and code will be publicly available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/17/2023

Syntactically Robust Training on Partially-Observed Data for Open Information Extraction

Open Information Extraction models have shown promising results with suf...
research
04/26/2022

PLOD: An Abbreviation Detection Dataset for Scientific Documents

The detection and extraction of abbreviations from unstructured texts ca...
research
07/14/2023

Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild

Cyber Threat Intelligence (CTI) plays a crucial role in assessing risks ...
research
02/20/2023

A Two-step Approach for Handling Zero-Cardinality in Relation Extraction

Relation tuple extraction from text is an important task for building kn...
research
09/14/2021

BenchIE: Open Information Extraction Evaluation Based on Facts, Not Tokens

Intrinsic evaluations of OIE systems are carried out either manually – w...
research
04/16/2021

Re-TACRED: Addressing Shortcomings of the TACRED Dataset

TACRED is one of the largest and most widely used sentence-level relatio...
research
05/11/2018

Neural Open Information Extraction

Conventional Open Information Extraction (Open IE) systems are usually b...

Please sign up or login with your details

Forgot password? Click here to reset