GAN2X: Non-Lambertian Inverse Rendering of Image GANs

by   Xingang Pan, et al.

2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components. Recovering these underlying intrinsic components from 2D images, also known as inverse rendering, usually requires a supervised setting with paired images collected from multiple viewpoints and lighting conditions, which is resource-demanding. In this work, we present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also recover non-Lambertian material properties by exploiting the pseudo paired data generated by a GAN. To achieve precise inverse rendering, we devise a specularity-aware neural surface representation that continuously models the geometry and material properties. A shading-based refinement technique is adopted to further distill information in the target image and recover more fine details. Experiments demonstrate that GAN2X can accurately decompose 2D images to 3D shape, albedo, and specular properties for different object categories, and achieves the state-of-the-art performance for unsupervised single-view 3D face reconstruction. We also show its applications in downstream tasks including real image editing and lifting 2D GANs to decomposed 3D GANs.


page 1

page 4

page 6

page 7

page 8

page 14

page 15

page 16


PANDORA: Polarization-Aided Neural Decomposition Of Radiance

Reconstructing an object's geometry and appearance from multiple images,...

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Natural images are projections of 3D objects on a 2D image plane. While ...

SupeRVol: Super-Resolution Shape and Reflectance Estimation in Inverse Volume Rendering

We propose an end-to-end inverse rendering pipeline called SupeRVol that...

De-rendering the World's Revolutionary Artefacts

Recent works have shown exciting results in unsupervised image de-render...

NeuS-PIR: Learning Relightable Neural Surface using Pre-Integrated Rendering

Recent advances in neural implicit fields enables rapidly reconstructing...

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

The advancement of generative radiance fields has pushed the boundary of...

End-to-end 3D shape inverse rendering of different classes of objects from a single input image

In this paper a semi-supervised deep framework is proposed for the probl...

Please sign up or login with your details

Forgot password? Click here to reset