Optimizing Search API Queries for Twitter Topic Classifiers Using a Maximum Set Coverage Approach

04/23/2019
by   Kasra Safari, et al.
0

Twitter has grown to become an important platform to access immediate information about major events and dynamic topics. As one example, recent work has shown that classifiers trained to detect topical content on Twitter can generalize well beyond the training data. Since access to Twitter data is hidden behind a limited search API, it is impossible (for most users) to apply these classifiers directly to the Twitter unfiltered data streams ("firehose"). Rather, applications must first decide what content to retrieve through the search API before filtering that content with topical classifiers. Thus, it is critically important to query the Twitter API relative to the intended topical classifier in a way that minimizes the amount of negatively classified data retrieved. In this paper, we propose a sequence of query optimization methods that generalize notions of the maximum coverage problem to find the subset of query terms within the API limits that cover most of the topically relevant tweets without sacrificing precision. We evaluate the proposed methods on a large dataset of Twitter data collected during 2013 and 2014 labeled using manually curated hashtags for eight topics. Among many insights, our analysis shows that the best of the proposed methods can significantly outperform the firehose on precision and F1-score while achieving high recall within strict API limitations.

READ FULL TEXT
research
10/08/2021

Smart Crawling: A New Approach toward Focus Crawling from Twitter

Twitter is a social network that offers a rich and interesting source of...
research
06/21/2020

Automatic Query Optimization for Retrieving Traffic Tweets

Twitter, like many social media and data brokering companies, makes thei...
research
05/11/2019

Mining Hidden Populations through Attributed Search

Researchers often query online social platforms through their applicatio...
research
03/27/2018

A Web Scraping Methodology for Bypassing Twitter API Restrictions

Retrieving information from social networks is the first and primordial ...
research
09/22/2022

Active Keyword Selection to Track Evolving Topics on Twitter

How can we study social interactions on evolving topics at a mass scale?...
research
07/21/2019

A Comprehensive Analysis of Twitter Trending Topics

Twitter is among the most used microblogging and online social networkin...
research
12/16/2020

You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection

Hate speech detection research has predominantly focused on purely conte...

Please sign up or login with your details

Forgot password? Click here to reset