LPT: Long-tailed Prompt Tuning for Image Classification

10/03/2022
by   Bowen Dong, et al.
5

For long-tailed classification, most works often pretrain a big model on a large-scale dataset, and then fine-tune the whole model for adapting to long-tailed data. Though promising, fine-tuning the whole pretrained model tends to suffer from high cost in computation and deployment of different models for different tasks, as well as weakened generalization ability for overfitting to certain features of long-tailed data. To alleviate these issues, we propose an effective Long-tailed Prompt Tuning method for long-tailed classification. LPT introduces several trainable prompts into a frozen pretrained model to adapt it to long-tailed data. For better effectiveness, we divide prompts into two groups: 1) a shared prompt for the whole long-tailed dataset to learn general features and to adapt a pretrained model into target domain; and 2) group-specific prompts to gather group-specific features for the samples which have similar features and also to empower the pretrained model with discrimination ability. Then we design a two-phase training paradigm to learn these prompts. In phase 1, we train the shared prompt via supervised prompt tuning to adapt a pretrained model to the desired long-tailed domain. In phase 2, we use the learnt shared prompt as query to select a small best matched set for a group of similar samples from the group-specific prompt set to dig the common features of these similar samples, then optimize these prompts with dual sampling strategy and asymmetric GCL loss. By only fine-tuning a few prompts while fixing the pretrained model, LPT can reduce training and deployment cost by storing a few prompts, and enjoys a strong generalization ability of the pretrained model. Experiments show that on various long-tailed benchmarks, with only  1.1 comparable performance than previous whole model fine-tuning methods, and is more robust to domain-shift.

READ FULL TEXT
research
03/09/2022

Domain Generalization using Pretrained Models without Fine-tuning

Fine-tuning pretrained models is a common practice in domain generalizat...
research
07/22/2022

Two-Stage Fine-Tuning: A Novel Strategy for Learning Class-Imbalanced Data

Classification on long-tailed distributed data is a challenging problem,...
research
09/18/2023

Parameter-Efficient Long-Tailed Recognition

The "pre-training and fine-tuning" paradigm in addressing long-tailed re...
research
10/13/2022

Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models

We introduce equi-tuning, a novel fine-tuning method that transforms (po...
research
09/05/2019

Effective Domain Knowledge Transfer with Soft Fine-tuning

Convolutional neural networks require numerous data for training. Consid...
research
01/14/2020

SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders

Knowing the similarity between sets of data has a number of positive imp...
research
04/20/2021

A novel three-stage training strategy for long-tailed classification

The long-tailed distribution datasets poses great challenges for deep le...

Please sign up or login with your details

Forgot password? Click here to reset