Multi-Modal Geometric Learning for Grasping and Manipulation

03/20/2018
by   Jacob Varley, et al.
0

This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object and tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

READ FULL TEXT

page 1

page 4

page 7

research
05/09/2022

Multi-Fingered In-Hand Manipulation with Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Multi-fingered hands could be used to achieve many dexterous manipulatio...
research
11/30/2022

Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds

The accurate detection and grasping of transparent objects are challengi...
research
05/19/2022

Action Conditioned Tactile Prediction: a case study on slip prediction

Tactile predictive models can be useful across several robotic manipulat...
research
05/08/2023

Rotational Slippage Prediction from Segmentation of Tactile Images

Adding tactile sensors to a robotic system is becoming a common practice...
research
04/29/2020

Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images

The connection between visual input and tactile sensing is critical for ...
research
02/23/2022

Visual-tactile sensing for Real-time liquid Volume Estimation in Grasping

We propose a deep visuo-tactile model for realtime estimation of the liq...
research
09/27/2019

Deep Gated Multi-modal Learning: In-hand Object Pose Estimation with Tactile and Image

In robot manipulation tasks, especially in-hand manipulation, estimation...

Please sign up or login with your details

Forgot password? Click here to reset