GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

04/15/2021
by   Zekun Hao, et al.
16

We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to estimate scene geometry and view-dependent appearance. In addition to camera trajectory, GANcraft allows user control over both scene semantics and output style. Experimental results with comparison to strong baselines show the effectiveness of GANcraft on this novel task of photorealistic 3D block world synthesis. The project website is available at https://nvlabs.github.io/GANcraft/ .

READ FULL TEXT

page 1

page 4

page 5

page 8

page 13

page 15

page 16

research
07/30/2020

Unsupervised Continuous Object Representation Networks for Novel View Synthesis

Novel View Synthesis (NVS) is concerned with the generation of novel vie...
research
02/28/2023

Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision

We address the challenging problem of jointly inferring the 3D flow and ...
research
10/24/2022

Learning Neural Radiance Fields from Multi-View Geometry

We present a framework, called MVG-NeRF, that combines classical Multi-V...
research
03/29/2022

Towards Learning Neural Representations from Shadows

We present a method that learns neural scene representations from only s...
research
10/24/2022

360-MLC: Multi-view Layout Consistency for Self-training and Hyper-parameter Tuning

We present 360-MLC, a self-training method based on multi-view layout co...
research
01/26/2023

Unsupervised Volumetric Animation

We propose a novel approach for unsupervised 3D animation of non-rigid d...
research
02/10/2022

Block-NeRF: Scalable Large Scene Neural View Synthesis

We present Block-NeRF, a variant of Neural Radiance Fields that can repr...

Please sign up or login with your details

Forgot password? Click here to reset