Feature-Proxy Transformer for Few-Shot Segmentation

by   Jian-Wei Zhang, et al.

Few-shot segmentation (FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the straightforward framework of "feature extractor + linear classification head" and proposes a novel Feature-Proxy Transformer (FPTrans) method, in which the "proxy" is the vector representing a semantic class in the linear classification head. FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from the bottom to top layers using a novel prompting strategy. 2) FPTrans uses multiple local background proxies (instead of a single one) because the background is not homogeneous and may contain some novel foreground regions. These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer. Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation. Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of-the-art decoder-based methods.


page 9

page 22


Few-Shot Segmentation via Cycle-Consistent Transformer

Few-shot segmentation aims to train a segmentation model that can fast a...

SimPropNet: Improved Similarity Propagation for Few-shot Image Segmentation

Few-shot segmentation (FSS) methods perform image segmentation for a par...

Query Semantic Reconstruction for Background in Few-Shot Segmentation

Few-shot segmentation (FSS) aims to segment unseen classes using a few a...

Task-Adaptive Feature Transformer with Semantic Enrichment for Few-Shot Segmentation

Few-shot learning allows machines to classify novel classes using only a...

Boosting Few-shot 3D Point Cloud Segmentation via Query-Guided Enhancement

Although extensive research has been conducted on 3D point cloud segment...

Multiscale Memory Comparator Transformer for Few-Shot Video Segmentation

Few-shot video segmentation is the task of delineating a specific novel ...

Doubly Deformable Aggregation of Covariance Matrices for Few-shot Segmentation

Training semantic segmentation models with few annotated samples has gre...

Please sign up or login with your details

Forgot password? Click here to reset