FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms

by   Henry B. Moss, et al.

We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify state-of-the-art performance from large collections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations based on single choices of random seeds. We show that reliable model selection also requires evaluations based on multiple train-test splits (contrary to common practice in many shared tasks). Using bandit theory from the statistics literature, we are able to adaptively determine appropriate numbers of data splits and random seeds used to evaluate each model, focusing computational resources on the evaluation of promising models whilst avoiding wasting evaluations on models with lower performance. Furthermore, our user-friendly Python implementation produces confidence guarantees of correctly selecting the optimal model. We evaluate our algorithms by selecting between 8 target-dependent sentiment analysis methods using dramatically fewer model evaluations than current model selection approaches.


page 1

page 2

page 3

page 4


Optimal subdata selection for linear model selection

If the assumed model does not accurately capture the underlying structur...

Multi-model mimicry for model selection according to generalised goodness-of-fit criteria

Selecting between candidate models is at the core of statistical practic...

Open Problem: Model Selection for Contextual Bandits

In statistical learning, algorithms for model selection allow the learne...

Model Selection Techniques -- An Overview

In the era of big data, analysts usually explore various statistical mod...

A Locally Adaptive Bayesian Cubature Method

Bayesian cubature (BC) is a popular inferential perspective on the cubat...

Model Selection for Ordinary Differential Equations: a Statistical Testing Approach

Ordinary differential equations (ODEs) are foundational in modeling intr...

Please sign up or login with your details

Forgot password? Click here to reset