Active Learning for Skewed Data Sets

by   Abbas Kazerouni, et al.

Consider a sequential active learning problem where, at each round, an agent selects a batch of unlabeled data points, queries their labels and updates a binary classifier. While there exists a rich body of work on active learning in this general form, in this paper, we focus on problems with two distinguishing characteristics: severe class imbalance (skew) and small amounts of initial training data. Both of these problems occur with surprising frequency in many web applications. For instance, detecting offensive or sensitive content in online communities (pornography, violence, and hate-speech) is receiving enormous attention from industry as well as research communities. Such problems have both the characteristics we describe – a vast majority of content is not offensive, so the number of positive examples for such content is orders of magnitude smaller than the negative examples. Furthermore, there is usually only a small amount of initial training data available when building machine-learned models to solve such problems. To address both these issues, we propose a hybrid active learning algorithm (HAL) that balances exploiting the knowledge available through the currently labeled training examples with exploring the large amount of unlabeled data available. Through simulation results, we show that HAL makes significantly better choices for what points to label when compared to strong baselines like margin-sampling. Classifiers trained on the examples selected for labeling by HAL easily out-perform the baselines on target metrics (like area under the precision-recall curve) given the same budget for labeling examples. We believe HAL offers a simple, intuitive, and computationally tractable way to structure active learning for a wide range of machine learning applications.


page 1

page 2

page 3

page 4


One-Round Active Learning

Active learning has been a main solution for reducing data labeling cost...

Data Shapley Valuation for Efficient Batch Active Learning

Annotating the right set of data amongst all available data points is a ...

Similarity Search for Efficient Active Learning and Search of Rare Concepts

Many active learning and search approaches are intractable for industria...

Batch Active Learning at Scale

The ability to train complex and highly effective models often requires ...

Zero-Round Active Learning

Active learning (AL) aims at reducing labeling effort by identifying the...

Exploiting Context for Robustness to Label Noise in Active Learning

Several works in computer vision have demonstrated the effectiveness of ...

ScatterShot: Interactive In-context Example Curation for Text Transformation

The in-context learning capabilities of LLMs like GPT-3 allow annotators...

Please sign up or login with your details

Forgot password? Click here to reset