MonoPerfCap: Human Performance Capture from Monocular Video

08/07/2017
by   Weipeng Xu, et al.
0

We present the first marker-less approach for temporally coherent 3D performance capture of a human with general clothing from monocular video. Our approach reconstructs articulated human skeleton motion as well as medium-scale non-rigid surface deformations in general scenes. Human performance capture is a challenging problem due to the large range of articulation, potentially fast motion, and considerable non-rigid deformations, even from multi-view data. Reconstruction from monocular video alone is drastically more challenging, since strong occlusions and the inherent depth ambiguity lead to a highly ill-posed reconstruction problem. We tackle these challenges by a novel approach that employs sparse 2D and 3D human pose detections from a convolutional neural network using a batch-based pose estimation strategy. Joint recovery of per-batch motion allows to resolve the ambiguities of the monocular reconstruction problem based on a low dimensional trajectory subspace. In addition, we propose refinement of the surface geometry based on fully automatically extracted silhouettes to enable medium-scale non-rigid alignment. We demonstrate state-of-the-art performance capture results that enable exciting applications such as video editing and free viewpoint video, previously infeasible from monocular video. Our qualitative and quantitative evaluation demonstrates that our approach significantly outperforms previous monocular methods in terms of accuracy, robustness and scene complexity that can be handled.

READ FULL TEXT

page 1

page 5

page 6

page 7

page 8

page 9

page 10

page 11

research
08/17/2022

MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes

3D human motion capture from monocular RGB images respecting interaction...
research
03/18/2020

DeepCap: Monocular Human Performance Capture Using Weak Supervision

Human performance capture is a highly important computer vision problem ...
research
06/16/2022

Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model

Capturing general deforming scenes is crucial for many computer graphics...
research
11/20/2021

A Deeper Look into DeepCap

Human performance capture is a highly important computer vision problem ...
research
10/05/2018

ReTiCaM: Real-time Human Performance Capture from Monocular Video

We present the first real-time human performance capture approach that r...
research
08/19/2023

Root Pose Decomposition Towards Generic Non-rigid 3D Reconstruction with Monocular Videos

This work focuses on the 3D reconstruction of non-rigid objects based on...
research
04/01/2021

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

We present a novel framework named NeuralRecon for real-time 3D scene re...

Please sign up or login with your details

Forgot password? Click here to reset