Hit ratio: An Evaluation Metric for Hashtag Recommendation

10/03/2020
by   Areej Alsini, et al.
0

Hashtag recommendation is a crucial task, especially with an increase of interest in using social media platforms such as Twitter in the last decade. Hashtag recommendation systems automatically suggest hashtags to a user while writing a tweet. Most of the research in the area of hashtag recommendation have used classical metrics such as hit rate, precision, recall, and F1-score to measure the accuracy of hashtag recommendation systems. These metrics are based on the exact match of the recommended hashtags with their corresponding ground truth. However, it is not clear how adequate these metrics to evaluate hashtag recommendation. The research question that we are interested in seeking an answer is: are these metrics adequate for evaluating hashtag recommendation systems when the numbers of ground truth hashtags in tweets are highly variable? In this paper, we propose a new metric which we call hit ratio for hashtag recommendation. Extensive evaluation through hypothetical examples and real-world application across a range of hashtag recommendation models indicate that the hit ratio is a useful metric. A comparison of hit ratio with the classical evaluation metrics reveals their limitations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/24/2023

#REVAL: a semantic evaluation framework for hashtag recommendation

Automatic evaluation of hashtag recommendation models is a fundamental t...
research
03/10/2023

Measuring and Detecting Virality on Social Media: The Case of Twitter's Viral Tweets Topic

Social media posts may go viral and reach large numbers of people within...
research
10/17/2019

Keyphrase Extraction from Disaster-related Tweets

While keyphrase extraction has received considerable attention in recent...
research
06/20/2021

On Sampling Top-K Recommendation Evaluation

Recently, Rendle has warned that the use of sampling-based top-k metrics...
research
07/31/2023

Metric@CustomerN: Evaluating Metrics at a Customer Level in E-Commerce

Accuracy measures such as Recall, Precision, and Hit Rate have been a st...
research
01/18/2014

Combining Evaluation Metrics via the Unanimous Improvement Ratio and its Application to Clustering Tasks

Many Artificial Intelligence tasks cannot be evaluated with a single qua...
research
03/02/2021

On Estimating Recommendation Evaluation Metrics under Sampling

Since the recent study (Krichene and Rendle 2020) done by Krichene and R...

Please sign up or login with your details

Forgot password? Click here to reset