DeepAI AI Chat
Log In Sign Up

3D GAN Inversion with Pose Optimization

by   Jaehoon Ko, et al.

With the recent advances in NeRF-based 3D aware GANs quality, projecting an image into the latent space of these 3D-aware GANs has a natural advantage over 2D GAN inversion: not only does it allow multi-view consistent editing of the projected image, but it also enables 3D reconstruction and novel view synthesis when given only a single image. However, the explicit viewpoint control acts as a main hindrance in the 3D GAN inversion process, as both camera pose and latent code have to be optimized simultaneously to reconstruct the given image. Most works that explore the latent space of the 3D-aware GANs rely on ground-truth camera viewpoint or deformable 3D model, thus limiting their applicability. In this work, we introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. The key to our approach is to leverage pre-trained estimators for better initialization and utilize the pixel-wise depth calculated from NeRF parameters to better reconstruct the given image. We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing to demonstrate the advantages of utilizing the latent space of 3D GANs. Additional results and visualizations are available at .


page 15

page 16

page 19

page 20

page 21

page 22

page 23

page 24


In-Domain GAN Inversion for Real Image Editing

Recent work has shown that a variety of controllable semantics emerges i...

3D GAN Inversion for Controllable Portrait Image Animation

Millions of images of human faces are captured every single day; but the...

LatentSwap3D: Semantic Edits on 3D Image GANs

Recent 3D-aware GANs rely on volumetric rendering techniques to disentan...

In-N-Out: Face Video Inversion and Editing with Volumetric Decomposition

3D-aware GANs offer new capabilities for creative content editing, such ...

FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold

Current Generative Adversarial Networks (GANs) produce photorealistic re...

Monocular 3D Object Reconstruction with GAN Inversion

Recovering a textured 3D mesh from a monocular image is highly challengi...

IMAGINE: Image Synthesis by Image-Guided Model Inversion

We introduce an inversion based method, denoted as IMAge-Guided model IN...