Prompt Algebra for Task Composition

06/01/2023
by   Pramuditha Perera, et al.
0

We investigate whether prompts learned independently for different tasks can be later combined through prompt algebra to obtain a model that supports composition of tasks. We consider Visual Language Models (VLM) with prompt tuning as our base classifier and formally define the notion of prompt algebra. We propose constrained prompt tuning to improve performance of the composite classifier. In the proposed scheme, prompts are constrained to appear in the lower dimensional subspace spanned by the basis vectors of the pre-trained vocabulary. Further regularization is added to ensure that the learned prompt is grounded correctly to the existing pre-trained vocabulary. We demonstrate the effectiveness of our method on object classification and object-attribute classification datasets. On average, our composite model obtains classification accuracy within 2.5 classification accuracy over the best base model by 8.45

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/18/2022

Task Residual for Tuning Vision-Language Models

Large-scale vision-language models (VLMs) pre-trained on billion-level d...
research
10/02/2019

The merits of Universal Language Model Fine-tuning for Small Datasets – a case with Dutch book reviews

We evaluated the effectiveness of using language models, that were pre-t...
research
07/16/2023

Tangent Model Composition for Ensembling and Continual Fine-tuning

Tangent Model Composition (TMC) is a method to combine component models ...
research
12/08/2022

Editing Models with Task Arithmetic

Changing how pre-trained models behave – e.g., improving their performan...
research
10/14/2021

Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models

Plug-and-play functionality allows deep learning models to adapt well to...
research
10/22/2019

Composite Neural Network: Theory and Application to PM2.5 Prediction

This work investigates the framework and performance issues of the compo...
research
05/11/2022

Making Pre-trained Language Models Good Long-tailed Learners

Prompt-tuning has shown appealing performance in few-shot classification...

Please sign up or login with your details

Forgot password? Click here to reset