DeepAI AI Chat
Log In Sign Up

A Pipeline for Post-Crisis Twitter Data Acquisition

by   Mayank Kejriwal, et al.
USC Information Sciences Institute
University of Southern California

Due to instant availability of data on social media platforms like Twitter, and advances in machine learning and data management technology, real-time crisis informatics has emerged as a prolific research area in the last decade. Although several benchmarks are now available, especially on portals like CrisisLex, an important, practical problem that has not been addressed thus far is the rapid acquisition and benchmarking of data from free, publicly available streams like the Twitter API. In this paper, we present ongoing work on a pipeline for facilitating immediate post-crisis data collection, curation and relevance filtering from the Twitter API. The pipeline is minimally supervised, alleviating the need for feature engineering by including a judicious mix of data preprocessing and fast text embeddings, along with an active learning framework. We illustrate the utility of the pipeline by describing a recent case study wherein it was used to collect and analyze millions of tweets in the immediate aftermath of the Las Vegas shootings.


The Pushshift Reddit Dataset

Social media data has become crucial to the advancement of scientific un...

A Web Scraping Methodology for Bypassing Twitter API Restrictions

Retrieving information from social networks is the first and primordial ...

Post or Tweet: Lessons from a Study of Facebook and Twitter Usage

This workshop paper reports on an ongoing mixed-methods study on the two...

Active Keyword Selection to Track Evolving Topics on Twitter

How can we study social interactions on evolving topics at a mass scale?...

Automatic Query Optimization for Retrieving Traffic Tweets

Twitter, like many social media and data brokering companies, makes thei...

Rapid-Rate: A Framework for Semi-supervised Real-time Sentiment Trend Detection in Unstructured Big Data

Commercial establishments like restaurants, service centres and retailer...