Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensor

by   Francois Robert Hogan, et al.

We introduce a new class of vision-based sensor and associated algorithmic processes that combine visual imaging with high-resolution tactile sending, all in a uniform hardware and computational architecture. We demonstrate the sensor's efficacy for both multi-modal object recognition and metrology. Object recognition is typically formulated as an unimodal task, but by combining two sensor modalities we show that we can achieve several significant performance improvements. This sensor, named the See-Through-your-Skin sensor (STS), is designed to provide rich multi-modal sensing of contact surfaces. Inspired by recent developments in optical tactile sensing technology, we address a key missing feature of these sensors: the ability to capture a visual perspective of the region beyond the contact surface. Whereas optical tactile sensors are typically opaque, we present a sensor with a semitransparent skin that has the dual capabilities of acting as a tactile sensor and/or as a visual camera depending on its internal lighting conditions. This paper details the design of the sensor, showcases its dual sensing capabilities, and presents a deep learning architecture that fuses vision and touch. We validate the ability of the sensor to classify household objects, recognize fine textures, and infer their physical properties both through numerical simulations and experiments with a smart countertop prototype.


page 2

page 3

page 5

page 7

page 8


TouchRoller: A Rolling Optical Tactile Sensor for Rapid Assessment of Large Surfaces

Tactile sensing is important for robots to perceive the world as it capt...

Learning Intuitive Physics with Multimodal Generative Models

Predicting the future interaction of objects when they come into contact...

GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger

This work describes the development of a high-resolution tactile-sensing...

Automatic breach detection during spine pedicle drilling based on vibroacoustic sensing

Pedicle drilling is a complex and critical spinal surgery task. Detectin...

ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition

Vision and touch are two of the important sensing modalities for humans ...

iCLAP: Shape Recognition by Combining Proprioception and Touch Sensing

For humans, both the proprioception and touch sensing are highly utilize...

Optical skin: Sensor-integration-free multimodal flexible sensing

The biological skin enables animals to sense various stimuli. Extensive ...

Please sign up or login with your details

Forgot password? Click here to reset