Online Active Model Selection for Pre-trained Classifiers

by   Mohammad Reza Karimi, et al.

Given k pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.


page 1

page 2

page 3

page 4


Cost-Effective Online Contextual Model Selection

How can we collect the most useful labels to learn a model selection pol...

Cost-Effective Training of Deep CNNs with Active Model Adaptation

Deep convolutional neural networks have achieved great success in variou...

Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning

Active learning is widely used to reduce labeling effort and training ti...

Active and Continuous Exploration with Deep Neural Networks and Expected Model Output Changes

The demands on visual recognition systems do not end with the complexity...

Active Learning for Skewed Data Sets

Consider a sequential active learning problem where, at each round, an a...

Stream-based active learning with linear models

The proliferation of automated data collection schemes and the advances ...

The Shapley Value of Classifiers in Ensemble Games

How do we decide the fair value of individual classifiers in an ensemble...