FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection

11/11/2020
by   Arda Inceoglu, et al.
0

Safe manipulation in unstructured environments for service robots is a challenging problem. A failure detection system is needed to monitor and detect unintended outcomes. We propose FINO-Net, a novel multimodal sensor fusion based deep neural network to detect and identify manipulation failures. We also introduce a multimodal dataset, containing 229 real-world manipulation data recorded with a Baxter robot. Our network combines RGB, depth and audio readings to effectively detect and classify failures. Results indicate that fusing RGB with depth and audio modalities significantly improves the performance. FINO-Net achieves 98.60 accuracy on our novel dataset. Code and data are publicly available at https://github.com/ardai/fino-net.

READ FULL TEXT

page 1

page 3

page 5

research
05/08/2023

Multimodal Detection and Identification of Robot Manipulation Failures

An autonomous service robot should be able to interact with its environm...
research
07/05/2021

Depth Quality-Inspired Feature Manipulation for Efficient RGB-D Salient Object Detection

RGB-D salient object detection (SOD) recently has attracted increasing r...
research
07/18/2017

Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments

Object detection is an essential task for autonomous robots operating in...
research
04/03/2022

Proactive Anomaly Detection for Robot Navigation with Multi-Sensor Fusion

Despite the rapid advancement of navigation algorithms, mobile robots of...
research
08/08/2022

Depth Quality-Inspired Feature Manipulation for Efficient RGB-D and Video Salient Object Detection

Recently CNN-based RGB-D salient object detection (SOD) has obtained sig...
research
02/29/2020

Robust Robotic Pouring using Audition and Haptics

Robust and accurate estimation of liquid height lies as an essential par...
research
09/19/2018

Music Mood Detection Based On Audio And Lyrics With Deep Neural Net

We consider the task of multimodal music mood prediction based on the au...

Please sign up or login with your details

Forgot password? Click here to reset