Active Testing: An Efficient and Robust Framework for Estimating Accuracy

07/02/2018
by   Phuc Nguyen, et al.
0

Much recent work on visual recognition aims to scale up learning to massive, noisily-annotated datasets. We address the problem of scaling- up the evaluation of such models to large-scale datasets with noisy labels. Current protocols for doing so require a human user to either vet (re-annotate) a small fraction of the test set and ignore the rest, or else correct errors in annotation as they are found through manual inspection of results. In this work, we re-formulate the problem as one of active testing, and examine strategies for efficiently querying a user so as to obtain an accu- rate performance estimate with minimal vetting. We demonstrate the effectiveness of our proposed active testing framework on estimating two performance metrics, Precision@K and mean Average Precision, for two popular computer vision tasks, multi-label classification and instance segmentation. We further show that our approach is able to save significant human annotation effort and is more robust than alternative evaluation protocols.

READ FULL TEXT
research
03/03/2022

Semantic-guided Image Virtual Attribute Learning for Noisy Multi-label Chest X-ray Classification

Deep learning methods have shown outstanding classification accuracy in ...
research
08/04/2016

Improving Multi-label Learning with Missing Labels by Structured Semantic Correlations

Multi-label learning has attracted significant interests in computer vis...
research
07/23/2022

Active Pointly-Supervised Instance Segmentation

The requirement of expensive annotations is a major burden for training ...
research
02/18/2022

Iterative Learning for Instance Segmentation

Instance segmentation is a computer vision task where separate objects i...
research
04/26/2021

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets

Data is the engine of modern computer vision, which necessitates collect...
research
05/31/2023

ActiveAED: A Human in the Loop Improves Annotation Error Detection

Manually annotated datasets are crucial for training and evaluating Natu...
research
12/26/2021

Budget Sensitive Reannotation of Noisy Relation Classification Data Using Label Hierarchy

Large crowd-sourced datasets are often noisy and relation classification...

Please sign up or login with your details

Forgot password? Click here to reset