TOCH: Spatio-Temporal Object Correspondence to Hand for Motion Refinement

by   Keyang Zhou, et al.
Max Planck Society
Universität Tübingen

We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. Existing hand trackers, especially those that rely on very few cameras, often produce visually unrealistic results with hand-object intersection or missing contacts. Although correcting such errors requires reasoning about temporal aspects of interaction, most previous work focus on static grasps and contacts. The core of our method are TOCH fields, a novel spatio-temporal representation for modeling correspondences between hands and objects during interaction. The key component is a point-wise object-centric representation which encodes the hand position relative to the object. Leveraging this novel representation, we learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art (SOTA) 3D hand-object interaction models, which are limited to static grasps and contacts. More importantly, our method produces smooth interactions even before and after contact. Using a single trained TOCH model, we quantitatively and qualitatively demonstrate its usefulness for 1) correcting erroneous reconstruction results from off-the-shelf RGB/RGB-D hand-object reconstruction methods, 2) de-noising, and 3) grasp transfer across objects. We will release our code and trained model on our project page at


page 2

page 3

page 10

page 12

page 13


Hand-Object Interaction Reasoning

This paper proposes an interaction reasoning network for modelling spati...

UV-Based 3D Hand-Object Reconstruction with Grasp Optimization

We propose a novel framework for 3D hand shape reconstruction and hand-o...

Articulated Objects in Free-form Hand Interaction

We use our hands to interact with and to manipulate objects. Articulated...

HOReeNet: 3D-aware Hand-Object Grasping Reenactment

We present HOReeNet, which tackles the novel task of manipulating images...

3D Object Reconstruction from Hand-Object Interactions

Recent advances have enabled 3d object reconstruction approaches using a...

FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow

Recent 4D shape representations model continuous temporal evolution of i...

Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

Hand motion capture is a popular research field, recently gaining more a...

Please sign up or login with your details

Forgot password? Click here to reset