Online Active Model Selection for Pre-trained Classifiers

10/19/2020
by   Mohammad Reza Karimi, et al.
0

Given k pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/13/2022

Cost-Effective Online Contextual Model Selection

How can we collect the most useful labels to learn a model selection pol...
02/15/2018

Cost-Effective Training of Deep CNNs with Active Model Adaptation

Deep convolutional neural networks have achieved great success in variou...
12/02/2020

Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning

Active learning is widely used to reduce labeling effort and training ti...
12/19/2016

Active and Continuous Exploration with Deep Neural Networks and Expected Model Output Changes

The demands on visual recognition systems do not end with the complexity...
05/23/2020

Active Learning for Skewed Data Sets

Consider a sequential active learning problem where, at each round, an a...
07/20/2022

Stream-based active learning with linear models

The proliferation of automated data collection schemes and the advances ...
01/06/2021

The Shapley Value of Classifiers in Ensemble Games

How do we decide the fair value of individual classifiers in an ensemble...