SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections

02/02/2023
by   Zhaoxi Chen, et al.
0

In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of SceneDreamer is a principled learning paradigm comprising 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of SceneDreamer and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds.

READ FULL TEXT

page 1

page 4

page 6

page 7

page 8

page 9

page 10

page 11

research
03/06/2023

Efficient Large-scale Scene Representation with a Hybrid of High-resolution Grid and Plane Features

Existing neural radiance fields (NeRF) methods for large-scale scene mod...
research
03/29/2022

Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images

Learning 3D generative models from a dataset of monocular images enables...
research
11/27/2022

3inGAN: Learning a 3D Generative Model from Images of a Self-similar Scene

We introduce 3inGAN, an unconditional 3D generative model trained from 2...
research
03/31/2021

RetrievalFuse: Neural 3D Scene Reconstruction with a Database

3D reconstruction of large scenes is a challenging problem due to the hi...
research
05/05/2022

BlobGAN: Spatially Disentangled Scene Representations

We propose an unsupervised, mid-level representation for a generative mo...
research
11/23/2021

Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

Though neural radiance fields (NeRF) have demonstrated impressive view s...
research
09/01/2023

CityDreamer: Compositional Generative Model of Unbounded 3D Cities

In recent years, extensive research has focused on 3D natural scene gene...

Please sign up or login with your details

Forgot password? Click here to reset