Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings

05/07/2019
by   Iva Gornishka, et al.
0

In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. We show these representations to be useful not only for categorizing users, but also for automatically generating user and community profiles. Inspired by traditional summarization approaches, we create the profiles by selecting diverse and representative content from all available modalities, i.e. the text, image and user modality. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations, to compare different embedding strategies, and to determine the importance of different modalities. We demonstrate the capabilities of the proposed approach on two different multimedia collections originating from the violent online extremism forum Stormfront and the microblogging platform Twitter, which are particularly interesting due to the high semantic level of the discussions they feature.

READ FULL TEXT
research
05/17/2019

Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks

There has been an explosion of multimodal content generated on social me...
research
12/01/2019

JNET: Learning User Representations via Joint Network Embedding and Topic Embedding

User representation learning is vital to capture diverse user preference...
research
04/18/2019

Exquisitor: Interactive Learning at Large

Increasing scale is a dominant trend in today's multimedia collections, ...
research
05/05/2020

II-20: Intelligent and pragmatic analytic categorization of image collections

We introduce II-20 (Image Insight 2020), a multimedia analytics approach...
research
09/12/2017

Multimodal Content Analysis for Effective Advertisements on YouTube

The rapid advances in e-commerce and Web 2.0 technologies have greatly i...
research
05/10/2023

Interpretable Multimodal Misinformation Detection with Logic Reasoning

Multimodal misinformation on online social platforms is becoming a criti...
research
07/06/2017

Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text

Real world multimedia data is often composed of multiple modalities such...

Please sign up or login with your details

Forgot password? Click here to reset