-
Question Answering on Freebase via Relation Extraction and Textual Evidence
Existing knowledge-based question answering systems often rely on small ...
read it
-
A Question-answering Based Framework for Relation Extraction Validation
Relation extraction is an important task in knowledge acquisition and te...
read it
-
TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task
TACRED (Zhang et al., 2017) is one of the largest, most widely used crow...
read it
-
Event Guided Denoising for Multilingual Relation Learning
General purpose relation extraction has recently seen considerable gains...
read it
-
On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
Many pairwise classification tasks, such as paraphrase detection and ope...
read it
-
Bootstrapping Relation Extractors using Syntactic Search by Examples
The advent of neural-networks in NLP brought with it substantial improve...
read it
-
Paired Examples as Indirect Supervision in Latent Decision Models
Compositional, structured models are appealing because they explicitly d...
read it
Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data
The process of collecting and annotating training data may introduce distribution artifacts which may limit the ability of models to learn correct generalization behavior. We identify failure modes of SOTA relation extraction (RE) models trained on TACRED, which we attribute to limitations in the data annotation process. We collect and annotate a challenge-set we call Challenging RE (CRE), based on naturally occurring corpus examples, to benchmark this behavior. Our experiments with four state-of-the-art RE models show that they have indeed adopted shallow heuristics that do not generalize to the challenge-set data. Further, we find that alternative question answering modeling performs significantly better than the SOTA models on the challenge-set, despite worse overall TACRED performance. By adding some of the challenge data as training examples, the performance of the model improves. Finally, we provide concrete suggestion on how to improve RE data collection to alleviate this behavior.
READ FULL TEXT
Comments
There are no comments yet.