Evaluation of Unsupervised Entity and Event Salience Estimation

by   Jiaying Lu, et al.

Salience Estimation aims to predict term importance in documents. Due to few existing human-annotated datasets and the subjective notion of salience, previous studies typically generate pseudo-ground truth for evaluation. However, our investigation reveals that the evaluation protocol proposed by prior work is difficult to replicate, thus leading to few follow-up studies existing. Moreover, the evaluation process is problematic: the entity linking tool used for entity matching is very noisy, while the ignorance of event argument for event evaluation leads to boosted performance. In this work, we propose a light yet practical entity and event salience estimation evaluation protocol, which incorporates the more reliable syntactic dependency parser. Furthermore, we conduct a comprehensive analysis among popular entity and event definition standards, and present our own definition for the Salience Estimation task to reduce noise during the pseudo-ground truth generation process. Furthermore, we construct dependency-based heterogeneous graphs to capture the interactions of entities and events. The empirical results show that both baseline methods and the novel GNN method utilizing the heterogeneous graph consistently outperform the previous SOTA model in all proposed metrics.


page 1

page 2

page 3

page 4


One for All: Neural Joint Modeling of Entities and Events

The previous work for event extraction has mainly focused on the predict...

Document-Level Multi-Event Extraction with Event Proxy Nodes and Hausdorff Distance Minimization

Document-level multi-event extraction aims to extract the structural inf...

Efficient Document-level Event Extraction via Pseudo-Trigger-aware Pruned Complete Graph

There are two main challenges in document-level event extraction: 1) arg...

'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities

Classic lexical-matching-based QA metrics are slowly being phased out be...

Iteratively Improving Biomedical Entity Linking and Event Extraction via Hard Expectation-Maximization

Biomedical entity linking and event extraction are two crucial tasks to ...

ELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool

We present Elevant, a tool for the fully automatic fine-grained evaluati...

Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer

Dependency graph, as a heterogeneous graph representing the intrinsic re...

Please sign up or login with your details

Forgot password? Click here to reset