Specular-to-Diffuse Translation for Multi-View Reconstruction

by   Shihao Wu, et al.

Most multi-view 3D reconstruction algorithms, especially when shape-from-shading cues are used, assume that object appearance is predominantly diffuse. To alleviate this restriction, we introduce S2Dnet, a generative adversarial network for transferring multiple views of objects with specular reflection into diffuse ones, so that multi-view reconstruction methods can be applied more effectively. Our network extends unsupervised image-to-image translation to multi-view "specular to diffuse" translation. To preserve object appearance across multiple views, we introduce a Multi-View Coherence loss (MVC) that evaluates the similarity and faithfulness of local patches after the view-transformation. Our MVC loss ensures that the similarity of local correspondences among multi-view images is preserved under the image-to-image translation. As a result, our network yields significantly better results than several single-view baseline techniques. In addition, we carefully design and generate a large synthetic training data set using physically-based rendering. During testing, our network takes only the raw glossy images as input, without extra information such as segmentation masks or lighting estimation. Results demonstrate that multi-view reconstruction can be significantly improved using the images filtered by our network. We also show promising performance on real world training and testing data.


page 2

page 11

page 12

page 13

page 14

page 17

page 18

page 19


Multi-View Image-to-Image Translation Supervised by 3D Pose

We address the task of multi-view image-to-image translation for person ...

Leveraging Multi-view Image Sets for Unsupervised Intrinsic Image Decomposition and Highlight Separation

We present an unsupervised approach for factorizing object appearance in...

Probabilistic Plant Modeling via Multi-View Image-to-Image Translation

This paper describes a method for inferring three-dimensional (3D) plant...

The importance of silhouette optimization in 3D shape reconstruction system from multiple object scenes

This paper presents a multi stage 3D shape reconstruction system of mult...

SiCloPe: Silhouette-Based Clothed People

We introduce a new silhouette-based representation for modeling clothed ...

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Inspired by the recent success of methods that employ shape priors to ac...

Higher Order Function Networks for View Planning and Multi-View Reconstruction

We consider the problem of planning views for a robot to acquire images ...

Please sign up or login with your details

Forgot password? Click here to reset