Analysis of Stopping Active Learning based on Stabilizing Predictions

04/23/2015
by   Michael Bloodgood, et al.
0

Within the natural language processing (NLP) community, active learning has been widely investigated and applied in order to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stopping active learning based on stabilizing predictions (SP). The analysis has revealed three elements that are central to the success of the SP method: (1) bounds on Cohen's Kappa agreement between successively trained models impose bounds on differences in F-measure performance of the models; (2) since the stop set does not have to be labeled, it can be made large in practice, helping to guarantee that the results transfer to previously unseen streams of examples at test/application time; and (3) good (low variance) sample estimates of Kappa between successive models can be obtained. Proofs of relationships between the level of Kappa agreement and the difference in performance between consecutive models are presented. Specifically, if the Kappa agreement between two models exceeds a threshold T (where T>0), then the difference in F-measure performance between those models is bounded above by 4(1-T)/T in all cases. If precision of the positive conjunction of the models is assumed to be p, then the bound can be tightened to 4(1-T)/(p+1)T.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/05/2021

Stopping Criterion for Active Learning Based on Error Stability

Active learning is a framework for supervised learning to improve the pr...
research
01/26/2019

Stopping Active Learning based on Predicted Change of F Measure for Text Classification

During active learning, an effective stopping method allows users to lim...
research
05/15/2020

Stopping criterion for active learning based on deterministic generalization bounds

Active learning is a framework in which the learning machine can select ...
research
01/08/2022

Impact of Stop Sets on Stopping Active Learning for Text Classification

Active learning is an increasingly important branch of machine learning ...
research
04/09/2015

Deciding when to stop: Efficient stopping of active learning guided drug-target prediction

Active learning has shown to reduce the number of experiments needed to ...
research
07/12/2018

How transferable are the datasets collected by active learners?

Active learning is a widely-used training strategy for maximizing predic...
research
12/22/2022

Investigating the agreement between methods of different precision

Agreement between measurement methods is commonly investigated by a so-c...

Please sign up or login with your details

Forgot password? Click here to reset