InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition

12/23/2021
by   Andreea Glavan, et al.
7

Indoor scene recognition is a growing field with great potential for behaviour understanding, robot localization, and elderly monitoring, among others. In this study, we approach the task of scene recognition from a novel standpoint, using multi-modal learning and video data gathered from social media. The accessibility and variety of social media videos can provide realistic data for modern scene recognition techniques and applications. We propose a model based on fusion of transcribed speech to text and visual features, which is used for classification on a novel dataset of social media videos of indoor scenes named InstaIndoor. Our model achieves up to 70 accuracy and 0.7 F1-Score. Furthermore, we highlight the potential of our approach by benchmarking on a YouTube-8M subset of indoor scenes as well, where it achieves 74 work pave the way to novel research in the challenging field of indoor scene recognition.

READ FULL TEXT

page 10

page 11

page 19

page 20

research
10/07/2019

Multi-Modal Machine Learning for Flood Detection in News, Social Media and Satellite Sequences

In this paper we present our methods for the MediaEval 2019 Mul-timedia ...
research
01/29/2016

What Can I Do Around Here? Deep Functional Scene Understanding for Cognitive Robots

For robots that have the capability to interact with the physical enviro...
research
05/06/2023

HateMM: A Multi-Modal Dataset for Hate Video Classification

Hate speech has become one of the most significant issues in modern soci...
research
03/25/2022

Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities

As social media platforms are evolving from text-based forums into multi...
research
07/02/2023

Fraunhofer SIT at CheckThat! 2023: Mixing Single-Modal Classifiers to Estimate the Check-Worthiness of Multi-Modal Tweets

The option of sharing images, videos and audio files on social media ope...
research
09/25/2021

An embarrassingly simple comparison of machine learning algorithms for indoor scene classification

With the emergence of autonomous indoor robots, the computer vision task...
research
08/03/2017

What your Facebook Profile Picture Reveals about your Personality

People spend considerable effort managing the impressions they give othe...

Please sign up or login with your details

Forgot password? Click here to reset