Learning to Synthesize Volumetric Meshes from Vision-based Tactile Imprints

03/29/2022
by   Xinghao Zhu, et al.
9

Vision-based tactile sensors typically utilize a deformable elastomer and a camera mounted above to provide high-resolution image observations of contacts. Obtaining accurate volumetric meshes for the deformed elastomer can provide direct contact information and benefit robotic grasping and manipulation. This paper focuses on learning to synthesize the volumetric mesh of the elastomer based on the image imprints acquired from vision-based tactile sensors. Synthetic image-mesh pairs and real-world images are gathered from 3D finite element methods (FEM) and physical sensors, respectively. A graph neural network (GNN) is introduced to learn the image-to-mesh mappings with supervised learning. A self-supervised adaptation method and image augmentation techniques are proposed to transfer networks from simulation to reality, from primitive contacts to unseen contacts, and from one sensor to another. Using these learned and adapted networks, our proposed method can accurately reconstruct the deformation of the real-world tactile sensor elastomer in various domains, as indicated by the quantitative and qualitative results.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 6

research
12/24/2020

Simulation of Vision-based Tactile Sensors using Physics based Rendering

Tactile sensing has seen a rapid adoption with the advent of vision-base...
research
12/21/2020

Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions

The images captured by vision-based tactile sensors carry information ab...
research
09/13/2023

GelFlow: Self-supervised Learning of Optical Flow for Vision-Based Tactile Sensor Displacement Measurement

High-resolution multi-modality information acquired by vision-based tact...
research
03/31/2021

Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections

Tactile sensing is critical for robotic grasping and manipulation of obj...
research
03/05/2020

Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing

Data-driven approaches to tactile sensing aim to overcome the complexity...
research
06/06/2020

Interpreting and Predicting Tactile Signals via a Physics-Based and Data-Driven Framework

High-density afferents in the human hand have long been regarded as esse...
research
05/21/2023

Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning

Recently, several morphologies, each with its advantages, have been prop...

Please sign up or login with your details

Forgot password? Click here to reset