Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion

03/15/2023
by   Inhwa Han, et al.
0

Diffusion models have shown superior performance in image generation and manipulation, but the inherent stochasticity presents challenges in preserving and manipulating image content and identity. While previous approaches like DreamBooth and Textual Inversion have proposed model or latent representation personalization to maintain the content, their reliance on multiple reference images and complex training limits their practicality. In this paper, we present a simple yet highly effective approach to personalization using highly personalized (HiPer) text embedding by decomposing the CLIP embedding space for personalization and content manipulation. Our method does not require model fine-tuning or identifiers, yet still enables manipulation of background, texture, and motion with just a single image and target text. Through experiments on diverse target texts, we demonstrate that our approach produces highly personalized and complex semantic image edits across a wide range of tasks. We believe that the novel understanding of the text embedding space presented in this work has the potential to inspire further research across various tasks.

READ FULL TEXT

page 3

page 5

page 6

page 8

page 11

page 12

page 13

page 14

research
10/05/2022

LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models

Research in vision-language models has seen rapid developments off-late,...
research
02/15/2023

PRedItOR: Text Guided Image Editing with Diffusion Prior

Diffusion models have shown remarkable capabilities in generating high q...
research
04/05/2023

Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models

This paper proposes a method for generating images of customized objects...
research
07/24/2021

Cycled Compositional Learning between Images and Text

We present an approach named the Cycled Composition Network that can mea...
research
07/13/2023

HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models

Personalization has emerged as a prominent aspect within the field of ge...
research
08/23/2023

Manipulating Embeddings of Stable Diffusion Prompts

Generative text-to-image models such as Stable Diffusion allow users to ...
research
12/22/2017

Disentangled Representations for Manipulation of Sentiment in Text

The ability to change arbitrary aspects of a text while leaving the core...

Please sign up or login with your details

Forgot password? Click here to reset