3D-aware Image Synthesis via Learning Structural and Textural Representations

by   Yinghao Xu, et al.
Zhejiang University
The Chinese University of Hong Kong

Making generative models 3D-aware bridges the 2D image space and the 3D physical world yet remains challenging. Recent attempts equip a Generative Adversarial Network (GAN) with a Neural Radiance Field (NeRF), which maps 3D coordinates to pixel values, as a 3D prior. However, the implicit function in NeRF has a very local receptive field, making the generator hard to become aware of the global structure. Meanwhile, NeRF is built on volume rendering which can be too costly to produce high-resolution results, increasing the optimization difficulty. To alleviate these two problems, we propose a novel framework, termed as VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation. We first learn a feature volume to represent the underlying structure, which is then converted to a feature field using a NeRF-like model. The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis. Such a design enables independent control of the shape and the appearance. Extensive experiments on a wide range of datasets show that our approach achieves sufficiently higher image quality and better 3D control than the previous methods.


page 1

page 3

page 5

page 6

page 8

page 12


pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

We have witnessed rapid progress on 3D-aware image synthesis, leveraging...

Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase

Despite the rapid advance of 3D-aware image synthesis, existing studies ...

High-Fidelity Synthesis with Disentangled Representation

Learning disentangled representation of data without supervision is an i...

Learning Compositional Radiance Fields of Dynamic Human Heads

Photorealistic rendering of dynamic humans is an important ability for t...

Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field

Recently 3D-aware GAN methods with neural radiance field have developed ...

Exemplar-bsaed Pattern Synthesis with Implicit Periodic Field Network

Synthesis of ergodic, stationary visual patterns is widely applicable in...

Real-Time Radiance Fields for Single-Image Portrait View Synthesis

We present a one-shot method to infer and render a photorealistic 3D rep...

Code Repositories


VolumeGAN - 3D-aware Image Synthesis via Learning Structural and Textural Representations

view repo

Please sign up or login with your details

Forgot password? Click here to reset