Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection

07/12/2022
by   Xubin Zhong, et al.
0

Human-Object Interaction (HOI) detection is a core task for high-level image understanding. Recently, Detection Transformer (DETR)-based HOI detectors have become popular due to their superior performance and efficient structure. However, these approaches typically adopt fixed HOI queries for all testing images, which is vulnerable to the location change of objects in one specific image. Accordingly, in this paper, we propose to enhance DETR's robustness by mining hard-positive queries, which are forced to make correct predictions using partial visual cues. First, we explicitly compose hard-positive queries according to the ground-truth (GT) position of labeled human-object pairs for each training image. Specifically, we shift the GT bounding boxes of each labeled human-object pair so that the shifted boxes cover only a certain portion of the GT ones. We encode the coordinates of the shifted boxes for each labeled human-object pair into an HOI query. Second, we implicitly construct another set of hard-positive queries by masking the top scores in cross-attention maps of the decoder layers. The masked attention maps then only cover partial important cues for HOI predictions. Finally, an alternate strategy is proposed that efficiently combines both types of hard queries. In each iteration, both DETR's learnable queries and one selected type of hard-positive queries are adopted for loss computation. Experimental results show that our proposed approach can be widely applied to existing DETR-based HOI detectors. Moreover, we consistently achieve state-of-the-art performance on three benchmarks: HICO-DET, V-COCO, and HOI-A. Code is available at https://github.com/MuchHair/HQM.

READ FULL TEXT

page 14

page 19

research
01/23/2022

Dynamic Label Assignment for Object Detection by Combining Predicted and Anchor IoUs

Label assignment plays a significant role in modern object detection mod...
research
06/13/2022

Featurized Query R-CNN

The query mechanism introduced in the DETR method is changing the paradi...
research
06/13/2022

Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection

Recent high-performing Human-Object Interaction (HOI) detection techniqu...
research
07/05/2023

Focusing on what to decode and what to train: Efficient Training with HOI Split Decoders and Specific Target Guided DeNoising

Recent one-stage transformer-based methods achieve notable gains in the ...
research
03/26/2022

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

The task of Human-Object Interaction (HOI) detection could be divided in...
research
09/21/2019

Generating Positive Bounding Boxes for Balanced Training of Object Detectors

Two-stage deep object detectors generate a set of regions-of-interest (R...
research
03/09/2021

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information

We propose a simple, intuitive yet powerful method for human-object inte...

Please sign up or login with your details

Forgot password? Click here to reset