Exploiting Multimodal Synthetic Data for Egocentric Human-Object Interaction Detection in an Industrial Scenario

by   Rosario Leonardi, et al.

In this paper, we tackle the problem of Egocentric Human-Object Interaction (EHOI) detection in an industrial setting. To overcome the lack of public datasets in this context, we propose a pipeline and a tool for generating synthetic images of EHOIs paired with several annotations and data signals (e.g., depth maps or instance segmentation masks). Using the proposed pipeline, we present EgoISM-HOI a new multimodal dataset composed of synthetic EHOI images in an industrial environment with rich annotations of hands and objects. To demonstrate the utility and effectiveness of synthetic EHOI data produced by the proposed tool, we designed a new method that predicts and combines different multimodal signals to detect EHOIs in RGB images. Our study shows that exploiting synthetic data to pre-train the proposed method significantly improves performance when tested on real-world data. Moreover, the proposed approach outperforms state-of-the-art class-agnostic methods. To support research in this field, we publicly release the datasets, source code, and pre-trained models at https://iplab.dmi.unict.it/egoism-hoi.


page 2

page 4

page 5

page 6

page 7

page 8

page 11


Egocentric Human-Object Interaction Detection Exploiting Synthetic Data

We consider the problem of detecting Egocentric HumanObject Interactions...

Seeing without Looking: Analysis Pipeline for Child Sexual Abuse Datasets

The online sharing and viewing of Child Sexual Abuse Material (CSAM) are...

MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain

Wearable cameras allow to acquire images and videos from the user's pers...

Segmenting unseen industrial components in a heavy clutter using rgb-d fusion and synthetic data

Segmentation of unseen industrial parts is essential for autonomous indu...

Learning to Segment Human Body Parts with Synthetically Trained Deep Convolutional Networks

This paper presents a new framework for human body part segmentation bas...

Synthetic 3D Data Generation Pipeline for Geometric Deep Learning in Architecture

With the growing interest in deep learning algorithms and computational ...

Please sign up or login with your details

Forgot password? Click here to reset