DeepAI AI Chat
Log In Sign Up

Identifying Wrongly Predicted Samples: A Method for Active Learning

by   Rahaf Aljundi, et al.

State-of-the-art machine learning models require access to significant amount of annotated data in order to achieve the desired level of performance. While unlabelled data can be largely available and even abundant, annotation process can be quite expensive and limiting. Under the assumption that some samples are more important for a given task than others, active learning targets the problem of identifying the most informative samples that one should acquire annotations for. Instead of the conventional reliance on model uncertainty as a proxy to leverage new unknown labels, in this work we propose a simple sample selection criterion that moves beyond uncertainty. By first accepting the model prediction and then judging its effect on the generalization error, we can better identify wrongly predicted samples. We further present an approximation to our criterion that is very efficient and provides a similarity based interpretation. In addition to evaluating our method on the standard benchmarks of active learning, we consider the challenging yet realistic scenario of imbalanced data where categories are not equally represented. We show state-of-the-art results and better rates at identifying wrongly predicted samples. Our method is simple, model agnostic and relies on the current model status without the need for re-training from scratch.


page 1

page 2

page 3

page 4


Influence Selection for Active Learning

The existing active learning methods select the samples by evaluating th...

Stopping Criterion for Active Learning Based on Error Stability

Active learning is a framework for supervised learning to improve the pr...

Temporal Output Discrepancy for Loss Estimation-based Active Learning

While deep learning succeeds in a wide range of tasks, it highly depends...

State-Relabeling Adversarial Active Learning

Active learning is to design label-efficient algorithms by sampling the ...

Robust and Discriminative Labeling for Multi-label Active Learning Based on Maximum Correntropy Criterion

Multi-label learning draws great interests in many real world applicatio...

Low-Regret Active learning

We develop an online learning algorithm for identifying unlabeled data p...

Man-recon: manifold learning for reconstruction with deep autoencoder for smart seismic interpretation

Deep learning can extract rich data representations if provided sufficie...