Joint Inference of Kinematic and Force Trajectories with Visuo-Tactile Sensing

by   Alexander Lambert, et al.

To perform complex tasks, robots must be able to interact with and manipulate their surroundings. One of the key challenges in accomplishing this is robust state estimation during physical interactions, where the state involves not only the robot and the object being manipulated, but also the state of the contact itself. In this work, within the context of planar pushing, we extend previous inference-based approaches to state estimation in several ways. We estimate the robot, object, and the contact state on multiple manipulation platforms configured with a vision-based articulated model tracker, and either a biomimetic tactile sensor or a force-torque sensor. We show how to fuse raw measurements from the tracker and tactile sensors to jointly estimate the trajectory of the kinematic states and the forces in the system via probabilistic inference on factor graphs, in both batch and incremental settings. We perform several benchmarks with our framework and show how performance is affected by incorporating various geometric and physics based constraints, occluding vision sensors, or injecting noise in tactile sensors. We also compare with prior work on multiple datasets and demonstrate that our approach can effectively optimize over multi-modal sensor data and reduce uncertainty to find better state estimates.


page 1

page 4

page 5

page 6


Visual Tactile Sensor Based Force Estimation for Position-Force Teleoperation

Vision-based tactile sensors have gained extensive attention in the robo...

In-Hand Object-Dynamics Inference using Tactile Fingertips

Having the ability to estimate an object's properties through interactio...

Realtime State Estimation with Tactile and Visual Sensing for Inserting a Suction-held Object

We develop a real-time state estimation system to recover the pose and c...

Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements

This paper addresses the localization of contacts of an unknown grasped ...

Learning to Predict Grip Quality from Simulation: Establishing a Digital Twin to Generate Simulated Data for a Grip Stability Metric

A robust grip is key to successful manipulation and joining of work piec...

Learning Tactile Models for Factor Graph-based State Estimation

We address the problem of estimating object pose from touch during manip...

Voronoi Features for Tactile Sensing: Direct Inference of Pressure, Shear, and Contact Locations

There are a wide range of features that tactile contact provides, each w...

Please sign up or login with your details

Forgot password? Click here to reset