Log In Sign Up

A Study on Evaluation Standard for Automatic Crack Detection Regard the Random Fractal

by   Hongyu Li, et al.

A reasonable evaluation standard underlies construction of effective deep learning models. However, we find in experiments that the automatic crack detectors based on deep learning are obviously underestimated by the widely used mean Average Precision (mAP) standard. This paper presents a study on the evaluation standard. It is clarified that the random fractal of crack disables the mAP standard, because the strict box matching in mAP calculation is unreasonable for the fractal feature. As a solution, a fractal-available evaluation standard named CovEval is proposed to correct the underestimation in crack detection. In CovEval, a different matching process based on the idea of covering box matching is adopted for this issue. In detail, Cover Area rate (CAr) is designed as a covering overlap, and a multi-match strategy is employed to release the one-to-one matching restriction in mAP. Extended Recall (XR), Extended Precision (XP) and Extended F-score (Fext) are defined for scoring the crack detectors. In experiments using several common frameworks for object detection, models get much higher scores in crack detection according to CovEval, which matches better with the visual performance. Moreover, based on faster R-CNN framework, we present a case study to optimize a crack detector based on CovEval standard. Recall (XR) of our best model achieves an industrial-level at 95.8, which implies that with reasonable standard for evaluation, the methods for object detection are with great potential for automatic industrial inspection.


page 1

page 3

page 4

page 6

page 7


Localization Recall Precision (LRP): A New Performance Metric for Object Detection

Average precision (AP), the area under the recall-precision (RP) curve, ...

Comparison of Object Detection Algorithms for Street-level Objects

Object detection for street-level objects can be applied to various use ...

How many radiographs are needed to re-train a deep learning system for object detection?

Background: Object detection in radiograph computer vision has largely b...

Sensitivity of Average Precision to Bounding Box Perturbations

Object detection is a fundamental vision task. It has been highly resear...

A Mask-RCNN Baseline for Probabilistic Object Detection

The Probabilistic Object Detection Challenge evaluates object detection ...

End-to-end training of object class detectors for mean average precision

We present a method for training CNN-based object class detectors direct...

One-Click Annotation with Guided Hierarchical Object Detection

The increase in data collection has made data annotation an interesting ...