Spatio-Temporal Learnable Proposals for End-to-End Video Object Detection

10/05/2022
by   Khurram Azeem Hashmi, et al.
0

This paper presents the novel idea of generating object proposals by leveraging temporal information for video object detection. The feature aggregation in modern region-based video object detectors heavily relies on learned proposals generated from a single-frame RPN. This imminently introduces additional components like NMS and produces unreliable proposals on low-quality frames. To tackle these restrictions, we present SparseVOD, a novel video object detection pipeline that employs Sparse R-CNN to exploit temporal information. In particular, we introduce two modules in the dynamic head of Sparse R-CNN. First, the Temporal Feature Extraction module based on the Temporal RoI Align operation is added to extract the RoI proposal features. Second, motivated by sequence-level semantic aggregation, we incorporate the attention-guided Semantic Proposal Feature Aggregation module to enhance object feature representation before detection. The proposed SparseVOD effectively alleviates the overhead of complicated post-processing methods and makes the overall pipeline end-to-end trainable. Extensive experiments show that our method significantly improves the single-frame Sparse RCNN by 8 Furthermore, besides achieving state-of-the-art 80.3 dataset with ResNet-50 backbone, our SparseVOD outperforms existing proposal-based methods by a significant margin on increasing IoU thresholds (IoU > 0.5).

READ FULL TEXT

page 1

page 2

page 5

page 8

research
07/11/2019

Object Detection in Video with Spatial-temporal Context Aggregation

Recent cutting-edge feature aggregation paradigms for video object detec...
research
09/08/2021

Temporal RoI Align for Video Object Recognition

Video object detection is challenging in the presence of appearance dete...
research
07/07/2022

Should All Proposals be Treated Equally in Object Detection?

The complexity-precision trade-off of an object detector is a critical p...
research
08/12/2021

TF-Blender: Temporal Feature Blender for Video Object Detection

Video objection detection is a challenging task because isolated video f...
research
07/15/2019

Sequence Level Semantics Aggregation for Video Object Detection

Video objection detection (VID) has been a rising research direction in ...
research
09/29/2022

4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation

In this work, we present a new paradigm, called 4D-StOP, to tackle the t...
research
10/02/2022

DFA: Dynamic Feature Aggregation for Efficient Video Object Detection

Video object detection is a fundamental yet challenging task in computer...

Please sign up or login with your details

Forgot password? Click here to reset