Efficient Multi-view Performance Capture of Fine-Scale Surface Detail

by   Nadia Robertini, et al.

We present a new effective way for performance capture of deforming meshes with fine-scale time-varying surface detail from multi-view video. Our method builds up on coarse 4D surface reconstructions, as obtained with commonly used template-based methods. As they only capture models of coarse-to-medium scale detail, fine scale deformation detail is often done in a second pass by using stereo constraints, features, or shading-based refinement. In this paper, we propose a new effective and stable solution to this second step. Our framework creates an implicit representation of the deformable mesh using a dense collection of 3D Gaussian functions on the surface, and a set of 2D Gaussians for the images. The fine scale deformation of all mesh vertices that maximizes photo-consistency can be efficiently found by densely optimizing a new model-to-image consistency energy on all vertex positions. A principal advantage is that our problem formulation yields a smooth closed form energy with implicit occlusion handling and analytic derivatives. Error-prone correspondence finding, or discrete sampling of surface displacement values are also not needed. We show several reconstructions of human subjects wearing loose clothing, and we qualitatively and quantitatively show that we robustly capture more detail than related methods.


page 1

page 4

page 6

page 7


Automatic 3D Reconstruction of Manifold Meshes via Delaunay Triangulation and Mesh Sweeping

In this paper we propose a new approach to incrementally initialize a ma...

Photometric Multi-View Mesh Refinement for High-Resolution Satellite Images

Modern high-resolution satellite sensors collect optical imagery with gr...

FastMESH: Fast Surface Reconstruction by Hexagonal Mesh-based Neural Rendering

Despite the promising results of multi-view reconstruction, the recent n...

Detail-preserving and Content-aware Variational Multi-view Stereo Reconstruction

Accurate recovery of 3D geometrical surfaces from calibrated 2D multi-vi...

Model-based Outdoor Performance Capture

We propose a new model-based method to accurately reconstruct human perf...

PiMPeR: Piecewise Dense 3D Reconstruction from Multi-View and Multi-Illumination Images

In this paper, we address the problem of dense 3D reconstruction from mu...

Mesh-based Camera Pairs Selection and Occlusion-Aware Masking for Mesh Refinement

Many Multi-View-Stereo algorithms extract a 3D mesh model of a scene, af...

Please sign up or login with your details

Forgot password? Click here to reset