Evaluation of Unsupervised Entity and Event Salience Estimation

by   Jiaying Lu, et al.

Salience Estimation aims to predict term importance in documents. Due to few existing human-annotated datasets and the subjective notion of salience, previous studies typically generate pseudo-ground truth for evaluation. However, our investigation reveals that the evaluation protocol proposed by prior work is difficult to replicate, thus leading to few follow-up studies existing. Moreover, the evaluation process is problematic: the entity linking tool used for entity matching is very noisy, while the ignorance of event argument for event evaluation leads to boosted performance. In this work, we propose a light yet practical entity and event salience estimation evaluation protocol, which incorporates the more reliable syntactic dependency parser. Furthermore, we conduct a comprehensive analysis among popular entity and event definition standards, and present our own definition for the Salience Estimation task to reduce noise during the pseudo-ground truth generation process. Furthermore, we construct dependency-based heterogeneous graphs to capture the interactions of entities and events. The empirical results show that both baseline methods and the novel GNN method utilizing the heterogeneous graph consistently outperform the previous SOTA model in all proposed metrics.


page 1

page 2

page 3

page 4


One for All: Neural Joint Modeling of Entities and Events

The previous work for event extraction has mainly focused on the predict...

Efficient Document-level Event Extraction via Pseudo-Trigger-aware Pruned Complete Graph

There are two main challenges in document-level event extraction: 1) arg...

'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities

Classic lexical-matching-based QA metrics are slowly being phased out be...

ELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool

We present Elevant, a tool for the fully automatic fine-grained evaluati...

Entity Embedding-based Anomaly Detection for Heterogeneous Categorical Events

Anomaly detection plays an important role in modern data-driven security...

Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer

Dependency graph, as a heterogeneous graph representing the intrinsic re...

Pilot Investigation for a Comprehensive Taxonomy of Autonomous Entities

This paper documents an exploratory pilot study to define the term Auton...