Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild

12/23/2020
by   Chung-Yi Weng, et al.
6

Given an "in-the-wild" video of a person, we reconstruct an animatable model of the person in the video. The output model can be rendered in any body pose to any camera view, via the learned controls, without explicit 3D mesh reconstruction. At the core of our method is a volumetric 3D human representation reconstructed with a deep network trained on input video, enabling novel pose/view synthesis. Our method is an advance over GAN-based image-to-image translation since it allows image synthesis for any pose and camera via the internal 3D representation, while at the same time it does not require a pre-rigged model or ground truth meshes for training, as in mesh-based learning. Experiments validate the design choices and yield results on synthetic data and on real videos of diverse people performing unconstrained activities (e.g. dancing or playing tennis). Finally, we demonstrate motion re-targeting and bullet-time rendering with the learned models.

READ FULL TEXT

page 1

page 4

page 6

page 7

page 8

research
11/24/2021

EgoRenderer: Rendering Human Avatars from Egocentric Camera Images

We present EgoRenderer, a system for rendering full-body neural avatars ...
research
12/04/2018

Monocular Total Capture: Posing Face, Body, and Hands in the Wild

We present the first method to capture the 3D total motion of a target p...
research
12/04/2018

Learning 3D Human Dynamics from Video

From an image of a person in action, we can easily guess the 3D motion o...
research
03/14/2019

Learning to Reconstruct People in Clothing from a Single RGB Camera

We present a learning-based model to infer the personalized 3D shape of ...
research
04/24/2023

Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis

We explore the task of embodied view synthesis from monocular videos of ...
research
12/31/2018

SiCloPe: Silhouette-Based Clothed People

We introduce a new silhouette-based representation for modeling clothed ...
research
01/14/2020

Neural Human Video Rendering: Joint Learning of Dynamic Textures and Rendering-to-Video Translation

Synthesizing realistic videos of humans using neural networks has been a...

Please sign up or login with your details

Forgot password? Click here to reset