S^2Contact: Graph-based Network for 3D Hand-Object Contact Estimation with Semi-Supervised Learning

08/01/2022
by   Tze Ho Elden Tse, et al.
0

Despite the recent efforts in accurate 3D annotations in hand and object datasets, there still exist gaps in 3D hand and object reconstructions. Existing works leverage contact maps to refine inaccurate hand-object pose estimations and generate grasps given object models. However, they require explicit 3D supervision which is seldom available and therefore, are limited to constrained settings, e.g., where thermal cameras observe residual heat left on manipulated objects. In this paper, we propose a novel semi-supervised framework that allows us to learn contact from monocular images. Specifically, we leverage visual and geometric consistency constraints in large-scale datasets for generating pseudo-labels in semi-supervised learning and propose an efficient graph-based network to infer contact. Our semi-supervised learning framework achieves a favourable improvement over the existing supervised learning methods trained on data with `limited' annotations. Notably, our proposed model is able to achieve superior results with less than half the network parameters and memory access cost when compared with the commonly-used PointNet-based approach. We show benefits from using a contact map that rules hand-object interactions to produce more accurate reconstructions. We further demonstrate that training with pseudo-labels can extend contact map estimations to out-of-domain objects and generalise better across multiple datasets.

READ FULL TEXT
research
10/19/2021

Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction

Every hand-object interaction begins with contact. Despite predicting th...
research
03/17/2022

DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detection

While numerous 3D detection works leverage the complementary relationshi...
research
06/02/2023

Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data

We propose UnMixMatch, a semi-supervised learning framework which can le...
research
05/17/2023

Confidence-Guided Semi-supervised Learning in Land Cover Classification

Semi-supervised learning has been well developed to help reduce the cost...
research
09/12/2019

Generating Accurate Pseudo-labels via Hermite Polynomials for SSL Confidently

Rectified Linear Units (ReLUs) are among the most widely used activation...
research
02/16/2019

Semi-supervised Learning on Graph with an Alternating Diffusion Process

Graph-based semi-supervised learning usually involves two separate stage...
research
03/11/2023

Semi-supervised Hand Appearance Recovery via Structure Disentanglement and Dual Adversarial Discrimination

Enormous hand images with reliable annotations are collected through mar...

Please sign up or login with your details

Forgot password? Click here to reset