DeepAI AI Chat
Log In Sign Up

Hitting the Target: Stopping Active Learning at the Cost-Based Optimum

10/07/2021
by   Zac Pullar-Strecker, et al.
University of Waikato
The University of Auckland
aucklanduni.ac.nz
1

Active learning allows machine learning models to be trained using fewer labels while retaining similar performance to traditional fully supervised learning. An active learner selects the most informative data points, requests their labels, and retrains itself. While this approach is promising, it leaves an open problem of how to determine when the model is `good enough' without the additional labels required for traditional evaluation. In the past, different stopping criteria have been proposed aiming to identify the optimal stopping point. However, optimality can only be expressed as a domain-dependent trade-off between accuracy and the number of labels, and no criterion is superior in all applications. This paper is the first to give actionable advice to practitioners on what stopping criteria they should use in a given real-world scenario. We contribute the first large-scale comparison of stopping criteria, using a cost measure to quantify the accuracy/label trade-off, public implementations of all stopping criteria we evaluate, and an open-source framework for evaluating stopping criteria. Our research enables practitioners to substantially reduce labelling costs by utilizing the stopping criterion which best suits their domain.

READ FULL TEXT

page 19

page 26

page 27

page 30

page 31

page 32

page 36

page 37

04/05/2021

Stopping Criterion for Active Learning Based on Error Stability

Active learning is a framework for supervised learning to improve the pr...
04/09/2015

Deciding when to stop: Efficient stopping of active learning guided drug-target prediction

Active learning has shown to reduce the number of experiments needed to ...
04/22/2022

Enough is Enough: Towards Autonomous Uncertainty-driven Stopping Criteria

Autonomous robotic exploration has long attracted the attention of the r...
06/19/2017

Unsure When to Stop? Ask Your Semantic Neighbors

In iterative supervised learning algorithms it is common to reach a poin...
09/13/2019

Modelling Stopping Criteria for Search Results using Poisson Processes

Text retrieval systems often return large sets of documents, particularl...
08/29/2021

Certifying One-Phase Technology-Assisted Reviews

Technology-assisted review (TAR) workflows based on iterative active lea...
05/15/2020

Stopping criterion for active learning based on deterministic generalization bounds

Active learning is a framework in which the learning machine can select ...