Magic3D: High-Resolution Text-to-3D Content Creation

11/18/2022
by   Chen-Hsuan Lin, et al.
0

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2x faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7 approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.

READ FULL TEXT

page 2

page 8

page 13

page 15

page 16

page 17

page 18

page 19

research
10/24/2022

High-Resolution Image Editing via Multi-Stage Blended Diffusion

Diffusion models have shown great results in image generation and in ima...
research
03/21/2023

3D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusion

We tackle the task of text-to-3D creation with pre-trained latent-based ...
research
06/16/2023

FALL-E: A Foley Sound Synthesis Model and Strategies

This paper introduces FALL-E, a foley synthesis system and its training/...
research
09/04/2023

Relay Diffusion: Unifying diffusion process across resolutions for image synthesis

Diffusion models achieved great success in image synthesis, but still fa...
research
05/10/2023

Text-guided High-definition Consistency Texture Model

With the advent of depth-to-image diffusion models, text-guided generati...
research
04/06/2023

DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Model

The increasing demand for high-quality 3D content creation has motivated...
research
04/24/2023

TextMesh: Generation of Realistic 3D Meshes From Text Prompts

The ability to generate highly realistic 2D images from mere text prompt...

Please sign up or login with your details

Forgot password? Click here to reset