HO-3D: A Multi-User, Multi-Object Dataset for Joint 3D Hand-Object Pose Estimation

by   Shreyas Hampali, et al.

We propose a new dataset for 3D hand+object pose estimation from color images, together with a method for efficiently annotating this dataset, and a 3D pose prediction method based on this dataset. The current lack of training data makes the 3D hand+object pose estimation very challenging. This lack is due to the complexity of labeling many real images with both 3D poses and of generating synthetic images with various realistic interaction. Moreover, even if synthetic images could be used for training, annotated real images are still needed for validation. To tackle this challenge, we capture sequences with a simple setup made of a single RGB-D camera. We also use a color camera imaging the sequences from a side view, but only for validation. We introduce a novel method based on global optimization that exploits depth, color, and temporal constraints for efficiently annotating the sequences, which we use to train another novel method that predicts both the 3D poses of the hand and the object from a single color image. Our hope is to encourage other researchers to develop better annotation methods for our dataset: One can then apply such method to capture and easily annotate sequences captured with a single RGB-D camera to easily create additional training data thus solving one of the main problems of 3D hand+object pose estimation.


page 2

page 5

page 8


Domain Transfer for 3D Pose Estimation from Color Images without Manual Annotations

We introduce a novel learning method for 3D pose estimation from color i...

Generating Realistic Training Images Based on Tonality-Alignment Generative Adversarial Networks for Hand Pose Estimation

Hand pose estimation from a monocular RGB image is an important but chal...

Bimodal Camera Pose Prediction for Endoscopy

Deducing the 3D structure of endoscopic scenes from images remains extre...

Estimating the Pose of a Euro Pallet with an RGB Camera based on Synthetic Training Data

Estimating the pose of a pallet and other logistics objects is crucial f...

ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis

Estimating the articulated 3D hand-object pose from a single RGB image i...

Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction

In this work, we study how well different type of approaches generalise ...

Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images

We propose a novel transformer-based framework that reconstructs two hig...

Please sign up or login with your details

Forgot password? Click here to reset