DeepAI AI Chat
Log In Sign Up

The Effect of Ground Truth Accuracy on the Evaluation of Localization Systems

by   Chen Gu, et al.

The ability to accurately evaluate the performance of location determination systems is crucial for many applications. Typically, the performance of such systems is obtained by comparing ground truth locations with estimated locations. However, these ground truth locations are usually obtained by clicking on a map or using other worldwide available technologies like GPS. This introduces ground truth errors that are due to the marking process, map distortions, or inherent GPS inaccuracy. In this paper, we present a theoretical framework for analyzing the effect of ground truth errors on the evaluation of localization systems. Based on that, we design two algorithms for computing the real algorithmic error from the validation error and marking/map ground truth errors, respectively. We further establish bounds on different performance metrics. Validation of our theoretical assumptions and analysis using real data collected in a typical environment shows the ability of our theoretical framework to correct the estimated error of a localization algorithm in the presence of ground truth errors. Specifically, our marking error algorithm matches the real error CDF within 4 more accurate estimate of the median/tail error by 150 shifted by 6m.


page 1

page 2


The Inconvenient Truths of Ground Truth for Binary Analysis

The effectiveness of binary analysis tools and techniques is often measu...

Deep Learning for Earth Image Segmentation based on Imperfect Polyline Labels with Annotation Errors

In recent years, deep learning techniques (e.g., U-Net, DeepLab) have ac...

Accurate Deep Direct Geo-Localization from Ground Imagery and Phone-Grade GPS

One of the most critical topics in autonomous driving or ride-sharing te...

Estimating regression errors without ground truth values

Regression analysis is a standard supervised machine learning method use...

Towards Reliable Evaluation of Road Network Reconstructions

Existing performance measures rank delineation algorithms inconsistently...

Automatic generation of ground truth for the evaluation of obstacle detection and tracking techniques

As automated vehicles are getting closer to becoming a reality, it will ...

Algebraic Ground Truth Inference: Non-Parametric Estimation of Sample Errors by AI Algorithms

Binary classification is widely used in ML production systems. Monitorin...