CrossRE: A Cross-Domain Dataset for Relation Extraction

10/17/2022
by   Elisa Bassignana, et al.
0

Relation Extraction (RE) has attracted increasing attention, but current RE evaluation is limited to in-domain evaluation setups. Little is known on how well a RE system fares in challenging, but realistic out-of-distribution evaluation setups. To address this gap, we propose CrossRE, a new, freely-available cross-domain benchmark for RE, which comprises six distinct text domains and includes multi-label annotations. An additional innovation is that we release meta-data collected during annotation, to include explanations and flags of difficult instances. We provide an empirical evaluation with a state-of-the-art model for relation classification. As the meta-data enables us to shed new light on the state-of-the-art model, we provide a comprehensive analysis on the impact of difficult cases and find correlations between model and human annotations. Overall, our empirical investigation highlights the difficulty of cross-domain RE. We release our dataset, to spur more research in this direction.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/28/2022

What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification

Over the last five years, research on Relation Extraction (RE) witnessed...
research
03/22/2015

Multilingual Open Relation Extraction Using Cross-lingual Projection

Open domain relation extraction systems identify relation and argument p...
research
07/07/2023

Linguistic representations for fewer-shot relation extraction across domains

Recent work has demonstrated the positive impact of incorporating lingui...
research
07/07/2019

Improving Cross-Domain Performance for Relation Extraction via Dependency Prediction and Information Flow Control

Relation Extraction (RE) is one of the fundamental tasks in Information ...
research
05/18/2023

Silver Syntax Pre-training for Cross-Domain Relation Extraction

Relation Extraction (RE) remains a challenging task, especially when con...
research
04/26/2020

MICK: A Meta-Learning Framework for Few-shot Relation Classification with Little Training Data

Few-shot relation classification describes a circumstance where a model ...
research
03/25/2022

A Cross-Domain Approach for Continuous Impression Recognition from Dyadic Audio-Visual-Physio Signals

The impression we make on others depends not only on what we say, but al...

Please sign up or login with your details

Forgot password? Click here to reset