Text-driven Visual Synthesis with Latent Diffusion Prior

02/16/2023
by   Ting-Hsuan Liao, et al.
0

There has been tremendous progress in large-scale text-to-image synthesis driven by diffusion models enabling versatile downstream applications such as 3D object synthesis from texts, image editing, and customized generation. We present a generic approach using latent diffusion models as powerful image priors for various visual synthesis tasks. Existing methods that utilize such priors fail to use these models' full capabilities. To improve this, our core ideas are 1) a feature matching loss between features from different layers of the decoder to provide detailed guidance and 2) a KL divergence loss to regularize the predicted latent features and stabilize the training. We demonstrate the efficacy of our approach on three different applications, text-to-3D, StyleGAN adaptation, and layered image editing. Extensive results show our method compares favorably against baselines.

READ FULL TEXT

page 1

page 6

page 7

page 8

research
09/29/2022

DreamFusion: Text-to-3D using 2D Diffusion

Recent breakthroughs in text-to-image synthesis have been driven by diff...
research
03/14/2023

Text-to-image Diffusion Model in Generative AI: A Survey

This survey reviews text-to-image diffusion models in the context that d...
research
07/24/2023

TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition

Text-driven diffusion models have exhibited impressive generative capabi...
research
06/06/2022

Blended Latent Diffusion

The tremendous progress in neural image generation, coupled with the eme...
research
07/30/2023

HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation

In this paper, we study Text-to-3D content generation leveraging 2D diff...
research
03/21/2023

Vox-E: Text-guided Voxel Editing of 3D Objects

Large scale text-guided diffusion models have garnered significant atten...
research
06/12/2023

Controlling Text-to-Image Diffusion by Orthogonal Finetuning

Large text-to-image diffusion models have impressive capabilities in gen...

Please sign up or login with your details

Forgot password? Click here to reset