Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation

02/14/2022
by   Jannik Kossen, et al.
3

We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach, whereas previous methods have focused on Monte Carlo estimates. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWING, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks. We further theoretically analyze ASEs' errors.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/09/2021

Active Testing: Sample-Efficient Model Evaluation

We introduce active testing: a new framework for sample-efficient model ...
research
11/16/2022

Data efficient surrogate modeling for engineering design: Ensemble-free batch mode deep active learning for regression

In a computer-aided engineering design optimization problem that involve...
research
09/30/2022

CEREAL: Few-Sample Clustering Evaluation

Evaluating clustering quality with reliable evaluation metrics like norm...
research
01/07/2023

Active Deep Learning Guided by Efficient Gaussian Process Surrogates

The success of active learning relies on the exploration of the underlyi...
research
11/19/2018

Deep Active Learning with a Neural Architecture Search

We consider active learning of deep neural networks. Most active learnin...
research
08/10/2021

Active Learning for Transition State Calculation

The transition state (TS) calculation is a grand challenge for computati...
research
06/03/2021

A generalized framework for active learning reliability: survey and benchmark

Active learning methods have recently surged in the literature due to th...

Please sign up or login with your details

Forgot password? Click here to reset