DeepAI AI Chat
Log In Sign Up

PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient

07/05/2022
by   Weihan Cao, et al.
ia.ac.cn
6

Knowledge distillation(KD) is a widely-used technique to train compact models in object detection. However, there is still a lack of study on how to distill between heterogeneous detectors. In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different. However, directly aligning the feature maps to distill detectors suffers from two problems. First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student. Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise. To address the above issues, we propose to imitate features with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features. Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs. Furthermore, it converges faster. With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5 the baseline, respectively.

READ FULL TEXT

page 2

page 3

page 15

page 16

page 17

07/12/2022

HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors

Conventional knowledge distillation (KD) methods for object detection ma...
03/26/2021

Distilling Object Detectors via Decoupled Features

Knowledge distillation is a widely used paradigm for inheriting informat...
06/23/2020

Distilling Object Detectors with Task Adaptive Regularization

Current state-of-the-art object detectors are at the expense of high com...
05/30/2022

Towards Efficient 3D Object Detection with Knowledge Distillation

Despite substantial progress in 3D object detection, advanced 3D detecto...
03/09/2023

Smooth and Stepwise Self-Distillation for Object Detection

Distilling the structured information captured in feature maps has contr...
09/23/2021

LGD: Label-guided Self-distillation for Object Detection

In this paper, we propose the first self-distillation framework for gene...
05/16/2018

Object detection at 200 Frames Per Second

In this paper, we propose an efficient and fast object detector which ca...