Optimizing Black-box Metrics with Iterative Example Weighting

02/18/2021
by   Gaurush Hiranandani, et al.
0

We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix. Such black-box learning settings are ubiquitous, for example, when the learner only has query access to the metric of interest, or in noisy-label and domain adaptation applications where the learner must evaluate the metric via performance evaluation using a small validation sample. Our approach is to adaptively learn example weights on the training dataset such that the resulting weighted objective best approximates the metric on the validation sample. We show how to model and estimate the example weights and use them to iteratively post-shift a pre-trained class probability estimator to construct a classifier. We also analyze the resulting procedure's statistical properties. Experiments on various label noise, domain shift, and fair classification setups confirm that our proposal is better than the individual state-of-the-art baselines for each application.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/27/2018

Metric-Optimized Example Weights

Real-world machine learning applications often have complex test metrics...
research
02/20/2020

Optimizing Black-box Metrics with Adaptive Surrogates

We address the problem of training models with black-box and hard-to-opt...
research
01/10/2020

Towards GAN Benchmarks Which Require Generalization

For many evaluation metrics commonly used as benchmarks for unconditiona...
research
04/21/2021

MetricOpt: Learning to Optimize Black-Box Evaluation Metrics

We study the problem of directly optimizing arbitrary non-differentiable...
research
02/12/2018

Detecting and Correcting for Label Shift with Black Box Predictors

Faced with distribution shift between training and test set, we wish to ...
research
08/02/2023

Curriculum Guided Domain Adaptation in the Dark

Addressing the rising concerns of privacy and security, domain adaptatio...
research
01/13/2023

Weighted RML using ensemble-methods for data assimilation

The weighting of critical-point samples in the weighted randomized maxim...

Please sign up or login with your details

Forgot password? Click here to reset