DeepAI
Log In Sign Up

Detecting Hands and Recognizing Physical Contact in the Wild

10/19/2020
by   Supreeth Narasimhaswamy, et al.
0

We investigate a new problem of detecting hands and recognizing their physical contact state in unconstrained conditions. This is a challenging inference task given the need to reason beyond the local appearance of hands. The lack of training annotations indicating which object or parts of an object the hand is in contact with further complicates the task. We propose a novel convolutional network based on Mask-RCNN that can jointly learn to localize hands and predict their physical contact to address this problem. The network uses outputs from another object detector to obtain locations of objects present in the scene. It uses these outputs and hand locations to recognize the hand's contact state using two attention mechanisms. The first attention mechanism is based on the hand and a region's affinity, enclosing the hand and the object, and densely pools features from this region to the hand region. The second attention module adaptively selects salient features from this plausible region of contact. To develop and evaluate our method's performance, we introduce a large-scale dataset called ContactHands, containing unconstrained images annotated with hand locations and contact states. The proposed network, including the parameters of attention modules, is end-to-end trainable. This network achieves approximately 7% relative improvement over a baseline network that was built on the vanilla Mask-RCNN architecture and trained for recognizing hand contact states.

READ FULL TEXT

page 3

page 6

page 9

04/09/2019

Contextual Attention for Hand Detection in the Wild

We present Hand-CNN, a novel convolutional network architecture for dete...
04/11/2019

Learning joint reconstruction of hands and manipulated objects

Estimating hand-object manipulations is essential for interpreting and i...
07/02/2021

HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset

HO-3D is a dataset providing image sequences of various hand-object inte...
02/28/2022

Background Mixup Data Augmentation for Hand and Object-in-Contact Detection

Detecting the positions of human hands and objects-in-contact (hand-obje...
06/11/2020

Understanding Human Hands in Contact at Internet Scale

Hands are the central means by which humans manipulate their world and b...
10/19/2021

Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction

Every hand-object interaction begins with contact. Despite predicting th...
12/06/2018

Context-Aware Synthesis and Placement of Object Instances

Learning to insert an object instance into an image in a semantically co...