Evaluation of Thematic Coherence in Microblogs

06/30/2021
by   Iman Munire Bilal, et al.
0

Collecting together microblogs representing opinions about the same topics within the same timeframe is useful to a number of different tasks and practitioners. A major question is how to evaluate the quality of such thematic clusters. Here we create a corpus of microblog clusters from three different domains and time windows and define the task of evaluating thematic coherence. We provide annotation guidelines and human annotations of thematic coherence by journalist experts. We subsequently investigate the efficacy of different automated evaluation metrics for the task. We consider a range of metrics including surface level metrics, ones for topic model coherence and text generation metrics (TGMs). While surface level metrics perform well, outperforming topic coherence metrics, they are not as consistent as TGMs. TGMs are more reliable than all other metrics considered for capturing thematic coherence in microblog clusters due to being less sensitive to the effect of time windows.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

05/18/2019

Automatic Evaluation of Local Topic Quality

Topic models are typically evaluated with respect to the global topic di...
10/08/2020

GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems

Automatically evaluating dialogue coherence is a challenging but high-de...
01/18/2017

First Study on Data Readiness Level

We introduce the idea of Data Readiness Level (DRL) to measure the relat...
07/05/2021

Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence

Topic model evaluation, like evaluation of other unsupervised methods, c...
04/06/2019

Evaluating Coherence in Dialogue Systems using Entailment

Evaluating open-domain dialogue systems is difficult due to the diversit...
07/04/2019

Multi-Task Learning for Coherence Modeling

We address the task of assessing discourse coherence, an aspect of text ...
10/26/2020

Melody Harmonization Using Orderless NADE, Chord Balancing, and Blocked Gibbs Sampling

Coherence and interestingness are two criteria for evaluating the perfor...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.