DeepAI AI Chat
Log In Sign Up

ATM: An Uncertainty-aware Active Self-training Framework for Label-efficient Text Classification

by   Yue Yu, et al.
Georgia Institute of Technology
University of Washington

Despite the great success of pre-trained language models (LMs) in many natural language processing (NLP) tasks, they require excessive labeled data for fine-tuning to achieve satisfactory performance. To enhance the label efficiency, researchers have resorted to active learning (AL), while the potential of unlabeled data is ignored by most of prior work. To unleash the power of unlabeled data for better label efficiency and model performance, we develop ATM, a new framework that leverage self-training to exploit unlabeled data and is agnostic to the specific AL algorithm, serving as a plug-in module to improve existing AL methods. Specifically, the unlabeled data with high uncertainty is exposed to oracle for annotations while those with low uncertainty are leveraged for self-training. To alleviate the label noise propagation issue in self-training, we design a simple and effective momentum-based memory bank to dynamically aggregate the model predictions from all rounds. By extensive experiments, we demonstrate that ATM outperforms the strongest active learning and self-training baselines and improve the label efficiency by 51.9


Bayesian Active Learning with Pretrained Language Models

Active Learning (AL) is a method to iteratively select data for annotati...

An Efficient Active Learning Pipeline for Legal Text Classification

Active Learning (AL) is a powerful tool for learning with less labeled d...

Uncertainty-aware Self-training for Text Classification with Few Labels

Recent success of large-scale pre-trained language models crucially hing...

Cold-start Active Learning through Self-supervised Language Modeling

Active learning strives to reduce annotation costs by choosing the most ...

Jasmine: A New Active Learning Approach to Combat Cybercrime

Over the past decade, the advent of cybercrime has accelarated the resea...

Deep Active Learning with Budget Annotation

Digital data collected over the decades and data currently being produce...

Active Learning of Non-semantic Speech Tasks with Pretrained Models

Pretraining neural networks with massive unlabeled datasets has become p...