Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation

03/20/2022
by   Zongyang Ma, et al.
9

Open-vocabulary object detection aims to detect novel object categories beyond the training set. The advanced open-vocabulary two-stage detectors employ instance-level visual-to-visual knowledge distillation to align the visual space of the detector with the semantic space of the Pre-trained Visual-Language Model (PVLM). However, in the more efficient one-stage detector, the absence of class-agnostic object proposals hinders the knowledge distillation on unseen objects, leading to severe performance degradation. In this paper, we propose a hierarchical visual-language knowledge distillation method, i.e., HierKD, for open-vocabulary one-stage detection. Specifically, a global-level knowledge distillation is explored to transfer the knowledge of unseen categories from the PVLM to the detector. Moreover, we combine the proposed global-level knowledge distillation and the common instance-level knowledge distillation to learn the knowledge of seen and unseen categories simultaneously. Extensive experiments on MS-COCO show that our method significantly surpasses the previous best one-stage detector with 11.9% and 6.7% AP_50 gains under the zero-shot detection and generalized zero-shot detection settings, and reduces the AP_50 performance gap from 14% to 7.3% compared to the best two-stage detector.

READ FULL TEXT
research
04/28/2021

Zero-Shot Detection via Vision and Language Knowledge Distillation

Zero-shot image classification has made promising progress by training t...
research
03/10/2023

Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection

Open-vocabulary object detection aims to provide object detectors traine...
research
09/30/2022

F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

We present F-VLM, a simple open-vocabulary object detection method built...
research
09/10/2023

Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels

In this paper, we investigate the task of zero-shot human-object interac...
research
01/23/2023

OvarNet: Towards Open-vocabulary Object Attribute Recognition

In this paper, we consider the problem of simultaneously detecting objec...
research
07/25/2022

Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations

While fine-tuning based methods for few-shot object detection have achie...
research
03/21/2023

Efficient Feature Distillation for Zero-shot Detection

The large-scale vision-language models (e.g., CLIP) are leveraged by dif...

Please sign up or login with your details

Forgot password? Click here to reset