Log In Sign Up

Diagnosing Errors in Video Relation Detectors

by   Shuo Chen, et al.

Video relation detection forms a new and challenging problem in computer vision, where subjects and objects need to be localized spatio-temporally and a predicate label needs to be assigned if and only if there is an interaction between the two. Despite recent progress in video relation detection, overall performance is still marginal and it remains unclear what the key factors are towards solving the problem. Following examples set in the object detection and action localization literature, we perform a deep dive into the error diagnosis of current video relation detection approaches. We introduce a diagnostic tool for analyzing the sources of detection errors. Our tool evaluates and compares current approaches beyond the single scalar metric of mean Average Precision by defining different error types specific to video relation detection, used for false positive analyses. Moreover, we examine different factors of influence on the performance in a false negative analysis, including relation length, number of subject/object/predicate instances, and subject/object size. Finally, we present the effect on video relation performance when considering an oracle fix for each error type. On two video relation benchmarks, we show where current approaches excel and fall short, allowing us to pinpoint the most important future directions in the field. The tool is available at <>.


page 3

page 5

page 6

page 7

page 8


Diagnosing Error in Temporal Action Detectors

Despite the recent progress in video understanding and the continuous ra...

Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors

Object detection remains as one of the most notorious open problems in c...

Localization Recall Precision (LRP): A New Performance Metric for Object Detection

Average precision (AP), the area under the recall-precision (RP) curve, ...

Video Relation Detection with Trajectory-aware Multi-modal Features

Video relation detection problem refers to the detection of the relation...

Analysing domain shift factors between videos and images for object detection

Object detection is one of the most important challenges in computer vis...

TIDE: A General Toolbox for Identifying Object Detection Errors

We introduce TIDE, a framework and associated toolbox for analyzing the ...

Optimal Correction Cost for Object Detection Evaluation

Mean Average Precision (mAP) is the primary evaluation measure for objec...