Capturing and Inferring Dense Full-Body Human-Scene Contact

06/20/2022
by   Chun-Hao P. Huang, et al.
14

Inferring human-scene contact (HSC) is the first step toward understanding how humans interact with their surroundings. While detecting 2D human-object interaction (HOI) and reconstructing 3D human pose and shape (HPS) have enjoyed significant progress, reasoning about 3D human-scene contact from a single image is still challenging. Existing HSC detection methods consider only a few types of predefined contact, often reduce body and scene to a small number of primitives, and even overlook image evidence. To predict human-scene contact from a single image, we address the limitations above from both data and algorithmic perspectives. We capture a new dataset called RICH for "Real scenes, Interaction, Contact and Humans." RICH contains multiview outdoor/indoor video sequences at 4K resolution, ground-truth 3D human bodies captured using markerless motion capture, 3D body scans, and high resolution 3D scene scans. A key feature of RICH is that it also contains accurate vertex-level contact labels on the body. Using RICH, we train a network that predicts dense body-scene contacts from a single RGB image. Our key insight is that regions in contact are always occluded so the network needs the ability to explore the whole image for evidence. We use a transformer to learn such non-local relationships and propose a new Body-Scene contact TRansfOrmer (BSTRO). Very few methods explore 3D contact; those that do focus on the feet only, detect foot contact as a post-processing step, or infer contact from body pose without looking at the scene. To our knowledge, BSTRO is the first method to directly estimate 3D body-scene contact from a single image. We demonstrate that BSTRO significantly outperforms the prior art. The code and dataset are available at https://rich.is.tue.mpg.de.

READ FULL TEXT

page 1

page 7

page 8

page 14

research
08/20/2019

Resolving 3D Human Pose Ambiguities with 3D Scene Constraints

To understand and analyze human behavior, we need to capture humans movi...
research
03/06/2023

Detecting Human-Object Contact in Images

Humans constantly contact objects to move and perform tasks. Thus, detec...
research
03/27/2023

Hi4D: 4D Instance Segmentation of Close Human Interaction

We propose Hi4D, a method and dataset for the automatic analysis of phys...
research
05/16/2023

Understanding 3D Object Interaction from a Single Image

Humans can easily understand a single image as depicting multiple potent...
research
08/03/2023

Reconstructing Three-Dimensional Models of Interacting Humans

Understanding 3d human interactions is fundamental for fine-grained scen...
research
12/18/2020

Learning Complex 3D Human Self-Contact

Monocular estimation of three dimensional human self-contact is fundamen...
research
12/04/2018

Walking on Thin Air: Environment-Free Physics-based Markerless Motion Capture

We propose a generative approach to physics-based motion capture. Unlike...

Please sign up or login with your details

Forgot password? Click here to reset