Deep Visuo-Tactile Learning: Estimation of Material Properties from Images

by   Kuniyuki Takahashi, et al.

Estimation of materials properties, such as softness or roughness from visual perception is an essential factor in deciding our way of interaction with our environment in e.g., object manipulation tasks or walking. In this research, we propose a method for deep visuo-tactile learning in which we train a encoder-decoder network with an intermediate layer in an unsupervised manner with images as input and tactile sequences as output. Materials properties are then represented in the intermediate layer as a continuous feature space and are estimated from image information. Unlike past studies utilizing tactile sensors focusing on classification for object recognition or recognizing material properties, our method does not require manually designing class labels or annotation, does not cause unknown objects to be classified into known discrete classes, and can be used without a tactile sensor after training. To collect data for training, we have attached a uSkin tactile sensor and a camera to the end-effector of a Sawyer robot to stroke surfaces of 21 different material surfaces. Our results after training show that features are indeed expressed continuously, and that our method is able to handle unknown objects in its feature space.


page 1

page 4

page 5


Active Clothing Material Perception using Tactile Sensing and Deep Learning

Humans represent the objects in the same category using their properties...

Design of a Biomimetic Tactile Sensor for Material Classification

Tactile sensing typically involves active exploration of unknown surface...

Simple Kinesthetic Haptics for Object Recognition

Object recognition is an essential capability when performing various ta...

Multimodal Zero-Shot Learning for Tactile Texture Recognition

Tactile sensing plays an irreplaceable role in robotic material recognit...

Robotic Tactile Perception of Object Properties: A Review

Touch sensing can help robots understand their sur- rounding environment...

Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images

The connection between visual input and tactile sensing is critical for ...

Learning efficient haptic shape exploration with a rigid tactile sensor array

Haptic exploration is a key skill for both robots and humans to discrimi...

Please sign up or login with your details

Forgot password? Click here to reset