Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

02/07/2023
by   Yuxin Wen, et al.
12

The strength of modern generative models lies in their ability to be controlled through text-based prompts. Typical "hard" prompts are made from interpretable words and tokens, and must be hand-crafted by humans. There are also "soft" prompts, which consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily interpreted, re-used across models, or plugged into a text-based interface. We describe an approach to robustly optimize hard text prompts through efficient gradient-based optimization. Our approach automatically generates hard text-based prompts for both text-to-image and text-to-text applications. In the text-to-image setting, the method creates hard prompts for diffusion models, allowing API users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model. In the text-to-text setting, we show that hard prompts can be automatically discovered that are effective in tuning LMs for classification.

READ FULL TEXT

page 1

page 4

page 5

page 6

page 7

page 13

research
12/28/2022

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models

Recent CLIP-guided 3D optimization methods, e.g., DreamFields and PureCL...
research
04/30/2018

optimParallel: an R Package Providing Parallel Versions of the Gradient-Based Optimization Methods of optim()

The R package optimParallel provides a parallel version of the gradient-...
research
06/08/2023

Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models

Text-to-image generative models have enabled high-resolution image synth...
research
08/03/2023

ConceptLab: Creative Generation using Diffusion Prior Constraints

Recent text-to-image generative models have enabled us to transform our ...
research
06/05/2023

Stable Diffusion is Unstable

Recently, text-to-image models have been thriving. Despite their powerfu...
research
03/01/2023

Selectively Hard Negative Mining for Alleviating Gradient Vanishing in Image-Text Matching

Recently, a series of Image-Text Matching (ITM) methods achieve impressi...
research
05/20/2019

Interpretable Neural Predictions with Differentiable Binary Variables

The success of neural networks comes hand in hand with a desire for more...

Please sign up or login with your details

Forgot password? Click here to reset