Hitting the Target: Stopping Active Learning at the Cost-Based Optimum

10/07/2021
by   Zac Pullar-Strecker, et al.
1

Active learning allows machine learning models to be trained using fewer labels while retaining similar performance to traditional fully supervised learning. An active learner selects the most informative data points, requests their labels, and retrains itself. While this approach is promising, it leaves an open problem of how to determine when the model is `good enough' without the additional labels required for traditional evaluation. In the past, different stopping criteria have been proposed aiming to identify the optimal stopping point. However, optimality can only be expressed as a domain-dependent trade-off between accuracy and the number of labels, and no criterion is superior in all applications. This paper is the first to give actionable advice to practitioners on what stopping criteria they should use in a given real-world scenario. We contribute the first large-scale comparison of stopping criteria, using a cost measure to quantify the accuracy/label trade-off, public implementations of all stopping criteria we evaluate, and an open-source framework for evaluating stopping criteria. Our research enables practitioners to substantially reduce labelling costs by utilizing the stopping criterion which best suits their domain.

READ FULL TEXT

page 19

page 26

page 27

page 30

page 31

page 32

page 36

page 37

research
04/05/2021

Stopping Criterion for Active Learning Based on Error Stability

Active learning is a framework for supervised learning to improve the pr...
research
04/09/2015

Deciding when to stop: Efficient stopping of active learning guided drug-target prediction

Active learning has shown to reduce the number of experiments needed to ...
research
04/22/2022

Enough is Enough: Towards Autonomous Uncertainty-driven Stopping Criteria

Autonomous robotic exploration has long attracted the attention of the r...
research
06/19/2017

Unsure When to Stop? Ask Your Semantic Neighbors

In iterative supervised learning algorithms it is common to reach a poin...
research
09/13/2019

Modelling Stopping Criteria for Search Results using Poisson Processes

Text retrieval systems often return large sets of documents, particularl...
research
08/29/2021

Certifying One-Phase Technology-Assisted Reviews

Technology-assisted review (TAR) workflows based on iterative active lea...
research
05/15/2020

Stopping criterion for active learning based on deterministic generalization bounds

Active learning is a framework in which the learning machine can select ...

Please sign up or login with your details

Forgot password? Click here to reset