Rectifying the Shortcut Learning of Background: Shared Object Concentration for Few-Shot Image Recognition

07/16/2021
by   Xu Luo, et al.
8

Few-Shot image classification aims to utilize pretrained knowledge learned from a large-scale dataset to tackle a series of downstream classification tasks. Typically, each task involves only few training examples from brand-new categories. This requires the pretraining models to focus on well-generalizable knowledge, but ignore domain-specific information. In this paper, we observe that image background serves as a source of domain-specific knowledge, which is a shortcut for models to learn in the source dataset, but is harmful when adapting to brand-new classes. To prevent the model from learning this shortcut knowledge, we propose COSOC, a novel Few-Shot Learning framework, to automatically figure out foreground objects at both pretraining and evaluation stage. COSOC is a two-stage algorithm motivated by the observation that foreground objects from different images within the same class share more similar patterns than backgrounds. At the pretraining stage, for each class, we cluster contrastive-pretrained features of randomly cropped image patches, such that crops containing only foreground objects can be identified by a single cluster. We then force the pretraining model to focus on found foreground objects by a fusion sampling strategy; at the evaluation stage, among images in each training class of any few-shot task, we seek for shared contents and filter out background. The recognized foreground objects of each class are used to match foreground of testing images. Extensive experiments tailored to inductive FSL tasks on two benchmarks demonstrate the state-of-the-art performance of our method.

READ FULL TEXT

page 8

page 15

page 16

page 19

page 20

page 21

page 22

page 23

research
03/22/2022

CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentation

Recent advances in self-supervised contrastive learning yield good image...
research
03/23/2023

The effectiveness of MAE pre-pretraining for billion-scale pretraining

This paper revisits the standard pretrain-then-finetune paradigm used in...
research
10/19/2022

Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models

Prompt learning is a new learning paradigm which reformulates downstream...
research
04/11/2022

Few-Shot Object Detection in Unseen Domains

Few-shot object detection (FSOD) has thrived in recent years to learn no...
research
04/16/2021

Semantic Image Matting

Natural image matting separates the foreground from background in fracti...
research
10/26/2021

Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysis

Aspect-based Sentiment Analysis (ABSA) aims to determine the sentiment p...
research
06/02/2023

Evaluating The Robustness of Self-Supervised Representations to Background/Foreground Removal

Despite impressive empirical advances of SSL in solving various tasks, t...

Please sign up or login with your details

Forgot password? Click here to reset