A Multimodal Framework for the Detection of Hateful Memes

12/23/2020
by   Phillip Lippe, et al.
0

An increasingly common expression of online hate speech is multimodal in nature and comes in the form of memes. Designing systems to automatically detect hateful content is of paramount importance if we are to mitigate its undesirable effects on the society at large. The detection of multimodal hate speech is an intrinsically difficult and open problem: memes convey a message using both images and text and, hence, require multimodal reasoning and joint visual and language understanding. In this work, we seek to advance this line of research and develop a multimodal framework for the detection of hateful memes. We improve the performance of existing multimodal approaches beyond simple fine-tuning and, among others, show the effectiveness of upsampling of contrastive examples to encourage multimodality and ensemble learning based on cross-validation to improve robustness. We furthermore analyze model misclassifications and discuss a number of hypothesis-driven augmentations and their effects on performance, presenting important implications for future research in the field. Our best approach comprises an ensemble of UNITER-based models and achieves an AUROC score of 80.53, placing us 4th on phase 2 of the 2020 Hateful Memes Challenge organized by Facebook.

READ FULL TEXT
research
10/09/2019

Exploring Hate Speech Detection in Multimodal Publications

In this work we target the problem of hate speech detection in multimoda...
research
07/25/2023

ARC-NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features

Text-embedded images can serve as a means of spreading hate speech, prop...
research
12/23/2020

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

Memes on the Internet are often harmless and sometimes amusing. However,...
research
12/20/2021

Hateful Memes Challenge: An Enhanced Multimodal Framework

Hateful Meme Challenge proposed by Facebook AI has attracted contestants...
research
05/10/2020

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

This work proposes a new challenge set for multimodal classification, fo...
research
04/14/2017

ShapeWorld - A new test methodology for multimodal language understanding

We introduce a novel framework for evaluating multimodal deep learning m...

Please sign up or login with your details

Forgot password? Click here to reset