StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model

03/16/2023
by   Zipeng Xu, et al.
0

Despite the progress made in the style transfer task, most previous work focus on transferring only relatively simple features like color or texture, while missing more abstract concepts such as overall art expression or painter-specific traits. However, these abstract semantics can be captured by models like DALL-E or CLIP, which have been trained using huge datasets of images and textual documents. In this paper, we propose StylerDALLE, a style transfer method that exploits both of these models and uses natural language to describe abstract art styles. Specifically, we formulate the language-guided style transfer task as a non-autoregressive token sequence translation, i.e., from input content image to output stylized image, in the discrete latent space of a large-scale pretrained vector-quantized tokenizer. To incorporate style information, we propose a Reinforcement Learning strategy with CLIP-based language supervision that ensures stylization and content preservation simultaneously. Experimental results demonstrate the superiority of our method, which can effectively transfer art styles using language instructions at different granularities. Code is available at https://github.com/zipengxuc/StylerDALLE.

READ FULL TEXT

page 6

page 15

page 16

page 17

page 18

page 19

page 20

page 21

research
02/28/2022

Name Your Style: An Arbitrary Artist-aware Image Style Transfer

Image style transfer has attracted widespread attention in the past few ...
research
06/26/2021

CAMS: Color-Aware Multi-Style Transfer

Image style transfer aims to manipulate the appearance of a source image...
research
03/06/2023

Neural Style Transfer for Vector Graphics

Neural style transfer draws researchers' attention, but the interest foc...
research
08/31/2022

Style-Agnostic Reinforcement Learning

We present a novel method of learning style-agnostic representation usin...
research
01/24/2023

Audience-Centric Natural Language Generation via Style Infusion

Adopting contextually appropriate, audience-tailored linguistic styles i...
research
08/16/2022

Language-guided Semantic Style Transfer of 3D Indoor Scenes

We address the new problem of language-guided semantic style transfer of...
research
11/20/2022

Font Representation Learning via Paired-glyph Matching

Fonts can convey profound meanings of words in various forms of glyphs. ...

Please sign up or login with your details

Forgot password? Click here to reset