DeepAI AI Chat
Log In Sign Up

Fast Model-Selection through Adapting Design of Experiments Maximizing Information Gain

by   Stefano Balietti, et al.
Northeastern University

To perform model-selection efficiently, we must run informative experiments. Here, we extend a seminal method for designing Bayesian optimal experiments that maximize the information gained from data collected. We introduce two computational improvements: a search algorithm from artificial intelligence and a sampling procedure shrinking the space of possible experiments to evaluate. We collected data for five different experimental designs and show that experiments optimized for information gain make model-selection faster, and cheaper, as compared to the designs chosen by a pool of expert experimentalists. Our procedure is general and can be applied iteratively to lab, field and online experiments.


page 4

page 9

page 13

page 23


Model Selection via the VC-Dimension

We derive an objective function that can be optimized to give an estimat...

Quasi-Maximum Likelihood based Model Selection Procedures for Binary Outcomes

In this paper, I propose two model selection procedures based on a quasi...

Better Model Selection with a new Definition of Feature Importance

Feature importance aims at measuring how crucial each input feature is f...

Online simulator-based experimental design for cognitive model selection

The problem of model selection with a limited number of experimental tri...

Model Selection for Production System via Automated Online Experiments

A challenge that machine learning practitioners in the industry face is ...

The numerical statistical fan for noisy experimental designs

Identifiability of polynomial models is a key requirement for multiple r...

Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes

Automating statistical modelling is a challenging problem that has far-r...