Parallel mesh reconstruction streams for pose estimation of interacting hands

04/25/2021
by   Uri Wollner, et al.
1

We present a new multi-stream 3D mesh reconstruction network (MSMR-Net) for hand pose estimation from a single RGB image. Our model consists of an image encoder followed by a mesh-convolution decoder composed of connected graph convolution layers. In contrast to previous models that form a single mesh decoding path, our decoder network incorporates multiple cross-resolution trajectories that are executed in parallel. Thus, global and local information are shared to form rich decoding representations at minor additional parameter cost compared to the single trajectory network. We demonstrate the effectiveness of our method in hand-hand and hand-object interaction scenarios at various levels of interaction. To evaluate the former scenario, we propose a method to generate RGB images of closely interacting hands. Moreoever, we suggest a metric to quantify the degree of interaction and show that close hand interactions are particularly challenging. Experimental results show that the MSMR-Net outperforms existing algorithms on the hand-object FreiHAND dataset as well as on our own hand-hand dataset.

READ FULL TEXT

page 3

page 4

page 8

page 13

page 14

page 15

research
08/21/2020

InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

Analysis of hand-hand interactions is a crucial step towards better unde...
research
09/29/2021

Understanding Egocentric Hand-Object Interactions from Hand Pose Estimation

In this paper, we address the problem of estimating the hand pose from t...
research
07/24/2021

Hand Image Understanding via Deep Multi-Task Learning

Analyzing and understanding hand information from multimedia materials l...
research
04/04/2020

Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild

We introduce a simple and effective network architecture for monocular 3...
research
04/27/2022

Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution

Estimating the pose and shape of hands and objects under interaction fin...
research
07/01/2021

Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation

In natural conversation and interaction, our hands often overlap or are ...
research
04/27/2023

A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Image

Recently, deep learning based approaches have shown promising results in...

Please sign up or login with your details

Forgot password? Click here to reset