Multimodal Categorization of Crisis Events in Social Media

by   Mahdi Abavisani, et al.

Recent developments in image classification and natural language processing, coupled with the rapid growth in social media usage, have enabled fundamental advances in detecting breaking events around the world in real-time. Emergency response is one such area that stands to gain from these advances. By processing billions of texts and images a minute, events can be automatically detected to enable emergency response workers to better assess rapidly evolving situations and deploy resources accordingly. To date, most event detection techniques in this area have focused on image-only or text-only approaches, limiting detection performance and impacting the quality of information delivered to crisis response teams. In this paper, we present a new multimodal fusion method that leverages both images and texts as input. In particular, we introduce a cross-attention module that can filter uninformative and misleading components from weak modalities on a sample by sample basis. In addition, we employ a multimodal graph-based approach to stochastically transition between embeddings of different multimodal pairs during training to better regularize the learning process as well as dealing with limited training data by constructing new matched pairs from different samples. We show that our method outperforms the unimodal approaches and strong multimodal baselines by a large margin on three crisis-related tasks.


page 1

page 2


Multimodal Classification of Events in Social Media

A large amount of social media hosted on platforms like Flickr and Insta...

Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response

Multimedia content in social media platforms provides significant inform...

Clustering of Social Media Messages for Humanitarian Aid Response during Crisis

Social media has quickly grown into an essential tool for people to comm...

CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and Summarization

Social media has increasingly played a key role in emergency response: f...

Deep Learning Benchmarks and Datasets for Social Media Image Classification for Disaster Response

During a disaster event, images shared on social media helps crisis mana...

Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises

The extensive use of social media platforms, especially during disasters...

Leveraging Personal Navigation Assistant Systems Using Automated Social Media Traffic Reporting

Modern urbanization is demanding smarter technologies to improve a varie...

Please sign up or login with your details

Forgot password? Click here to reset