DeepAI
Log In Sign Up

Model Selection for Production System via Automated Online Experiments

05/27/2021
by   Zhenwen Dai, et al.
0

A challenge that machine learning practitioners in the industry face is the task of selecting the best model to deploy in production. As a model is often an intermediate component of a production system, online controlled experiments such as A/B tests yield the most reliable estimation of the effectiveness of the whole system, but can only compare two or a few models due to budget constraints. We propose an automated online experimentation mechanism that can efficiently perform model selection from a large pool of models with a small number of online experiments. We derive the probability distribution of the metric of interest that contains the model uncertainty from our Bayesian surrogate model trained using historical logs. Our method efficiently identifies the best model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation. Using simulations based on real data, we demonstrate the effectiveness of our method on two different tasks.

READ FULL TEXT

page 7

page 17

page 18

11/21/2019

Multi-model mimicry for model selection according to generalised goodness-of-fit criteria

Selecting between candidate models is at the core of statistical practic...
07/13/2022

Cost-Effective Online Contextual Model Selection

How can we collect the most useful labels to learn a model selection pol...
01/23/2013

On Supervised Selection of Bayesian Networks

Given a set of possible models (e.g., Bayesian network structures) and a...
07/10/2018

Fast Model-Selection through Adapting Design of Experiments Maximizing Information Gain

To perform model-selection efficiently, we must run informative experime...
02/07/2019

Model Selection for Simulator-based Statistical Models: A Kernel Approach

We propose a novel approach to model selection for simulator-based stati...
12/24/2019

Meta-Learning PAC-Bayes Priors in Model Averaging

Nowadays model uncertainty has become one of the most important problems...