Generating Person-Scene Interactions in 3D Scenes

by   Siwei Zhang, et al.

High fidelity digital 3D environments have been proposed in recent years; however, it remains extreme challenging to automatically equip such environment with realistic human bodies. Existing work utilizes images, depths, or semantic maps to represent the scene, and parametric human models to represent 3D bodies in the scene. While being straightforward, their generated human-scene interactions are often lack of naturalness and physical plausibility. Our key observation is that humans interact with the world through body-scene contact. To explicitly and effectively represent the physical contact between the body and the world is essential for modeling human-scene interaction. To that end, we propose a novel interaction representation, which explicitly encodes the proximity between the human body and the 3D scene around it. Specifically, given a set of basis points on a scene mesh, we leverage a conditional variational autoencoder to synthesize the distance from every basis point to its closest point on a human body. The synthesized proximal relationship between the human body and the scene can indicate which region a person tends to contact. Furthermore, based on such synthesized proximity, we can effectively obtain expressive 3D human bodies that naturally interact with the 3D scene. Our perceptual study shows that our model significantly improves the state-of-the-art method, approaching the realism of real human-scene interaction. We believe our method makes an important step towards the fully automatic synthesis of realistic 3D human bodies in 3D scenes. Our code and model will be publicly available for research purpose.



There are no comments yet.


page 1

page 4

page 5

page 7

page 8

page 14

page 15

page 16


Populating 3D Scenes by Learning Human-Scene Interaction

Humans live within a 3D space and constantly interact with it to perform...

Digital Voodoo Dolls

An institution, be it a body of government, commercial enterprise, or a ...

Generating 3D People in Scenes without People

We present a fully-automatic system that takes a 3D scene and generates ...

The Wanderings of Odysseus in 3D Scenes

Our goal is to populate digital environments, in which the digital human...

Binge Watching: Scaling Affordance Learning from Sitcoms

In recent years, there has been a renewed interest in jointly modeling p...

Stochastic Scene-Aware Motion Prediction

A long-standing goal in computer vision is to capture, model, and realis...

3D Reconstruction of Crime Scenes and Design Considerations for an Interactive Investigation Tool

Crime Scene Investigation (CSI) is a carefully planned systematic proces...

Code Repositories


Official Pytorch implementation for 2020 3DV paper "PLACE: Proximity Learning of Articulation and Contact in 3D Environments" and trained models

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.