IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors

10/07/2022
by   Sheng Xu, et al.
0

Knowledge distillation (KD) has been proven to be useful for training compact object detection models. However, we observe that KD is often effective when the teacher model and student counterpart share similar proposal information. This explains why existing KD methods are less effective for 1-bit detectors, caused by a significant information discrepancy between the real-valued teacher and the 1-bit student. This paper presents an Information Discrepancy-aware strategy (IDa-Det) to distill 1-bit detectors that can effectively eliminate information discrepancies and significantly reduce the performance gap between a 1-bit detector and its real-valued counterpart. We formulate the distillation process as a bi-level optimization formulation. At the inner level, we select the representative proposals with maximum information discrepancy. We then introduce a novel entropy distillation loss to reduce the disparity based on the selected proposals. Extensive experiments demonstrate IDa-Det's superiority over state-of-the-art 1-bit detectors and KD methods on both PASCAL VOC and COCO datasets. IDa-Det achieves a 76.9 ResNet-18 backbone. Our code is open-sourced on https://github.com/SteveTsui/IDa-Det.

READ FULL TEXT
research
08/20/2023

Representation Disparity-aware Distillation for 3D Object Detection

In this paper, we focus on developing knowledge distillation (KD) for co...
research
03/26/2021

Distilling Object Detectors via Decoupled Features

Knowledge distillation is a widely used paradigm for inheriting informat...
research
06/09/2019

Distilling Object Detectors with Fine-grained Feature Imitation

State-of-the-art CNN based recognition models are often computationally ...
research
04/01/2023

Q-DETR: An Efficient Low-Bit Quantized Detection Transformer

The recent detection transformer (DETR) has advanced object detection, b...
research
07/12/2022

HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors

Conventional knowledge distillation (KD) methods for object detection ma...
research
12/31/2022

Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching

Considering the computation complexity, we propose a Guided Hybrid Quant...
research
09/27/2021

Deep Structured Instance Graph for Distilling Object Detectors

Effectively structuring deep knowledge plays a pivotal role in transfer ...

Please sign up or login with your details

Forgot password? Click here to reset