FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow

by   Jihyun Lee, et al.

Recent 4D shape representations model continuous temporal evolution of implicit shapes by (1) learning query flows without leveraging shape and articulation priors or (2) decoding shape occupancies separately for each time value. Thus, they do not effectively capture implicit correspondences between articulated shapes or regularize jittery temporal deformations. In this work, we present FourierHandFlow, which is a spatio-temporally continuous representation for human hands that combines a 3D occupancy field with articulation-aware query flows represented as Fourier series. Given an input RGB sequence, we aim to learn a fixed number of Fourier coefficients for each query flow to guarantee smooth and continuous temporal shape dynamics. To effectively model spatio-temporal deformations of articulated hands, we compose our 4D representation based on two types of Fourier query flow: (1) pose flow that models query dynamics influenced by hand articulation changes via implicit linear blend skinning and (2) shape flow that models query-wise displacement flow. In the experiments, our method achieves state-of-the-art results on video-based 4D reconstruction while being computationally more efficient than the existing 3D/4D implicit shape representations. We additionally show our results on motion inter- and extrapolation and texture transfer using the learned correspondences of implicit shapes. To the best of our knowledge, FourierHandFlow is the first neural 4D continuous hand representation learned from RGB videos. The code will be publicly accessible.


page 1

page 8

page 9


Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes

We present Implicit Two Hands (Im2Hands), the first neural implicit repr...

Spatio-temporal motion completion using a sequence of latent primitives

We propose a markerless performance capture method that computes a tempo...

Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstruction

This paper focuses on the task of 4D shape reconstruction from a sequenc...

Deep Implicit Templates for 3D Shape Representation

Deep implicit functions (DIFs), as a kind of 3D shape representation, ar...

TOCH: Spatio-Temporal Object Correspondence to Hand for Motion Refinement

We present TOCH, a method for refining incorrect 3D hand-object interact...

Learning Video Representations from Correspondence Proposals

Correspondences between frames encode rich information about dynamic con...

Interactive Exploration of the Temporal α-Shape

An interesting subcomplex of the Delaunay triangulation are α-shapes, wh...

Please sign up or login with your details

Forgot password? Click here to reset