Debiasing Scores and Prompts of 2D Diffusion for Robust Text-to-3D Generation

03/27/2023
by   Susung Hong, et al.
0

The view inconsistency problem in score-distilling text-to-3D generation, also known as the Janus problem, arises from the intrinsic bias of 2D diffusion models, which leads to the unrealistic generation of 3D objects. In this work, we explore score-distilling text-to-3D generation and identify the main causes of the Janus problem. Based on these findings, we propose two approaches to debias the score-distillation frameworks for robust text-to-3D generation. Our first approach, called score debiasing, involves gradually increasing the truncation value for the score estimated by 2D diffusion models throughout the optimization process. Our second approach, called prompt debiasing, identifies conflicting words between user prompts and view prompts utilizing a language model and adjusts the discrepancy between view prompts and object-space camera poses. Our experimental results show that our methods improve realism by significantly reducing artifacts and achieve a good trade-off between faithfulness to the 2D diffusion models and 3D consistency with little overhead.

READ FULL TEXT

page 1

page 3

page 4

page 6

page 9

page 10

research
03/14/2023

Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation

Text-to-3D generation has shown rapid progress in recent days with the a...
research
05/25/2023

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Score distillation sampling (SDS) has shown great promise in text-to-3D ...
research
05/06/2023

Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation

Recently, continuous diffusion models (CDM) have been introduced into no...
research
03/30/2021

AfriKI: Machine-in-the-Loop Afrikaans Poetry Generation

This paper proposes a generative language model called AfriKI. Our appro...
research
12/01/2022

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

A diffusion model learns to predict a vector field of gradients. We prop...
research
09/07/2023

Chasing Consistency in Text-to-3D Generation from a Single Image

Text-to-3D generation from a single-view image is a popular but challeng...
research
05/08/2023

Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!

Diffusion models have been successfully adapted to text generation tasks...

Please sign up or login with your details

Forgot password? Click here to reset