DECoVaC: Design of Experiments with Controlled Variability Components

09/21/2019
by   Thomas Boquet, et al.
0

Reproducible research in Machine Learning has seen a salutary abundance of progress lately: workflows, transparency, and statistical analysis of validation and test performance. We build on these efforts and take them further. We offer a principled experimental design methodology, based on linear mixed models, to study and separate the effects of multiple factors of variation in machine learning experiments. This approach allows to account for the effects of architecture, optimizer, hyper-parameters, intentional randomization, as well as unintended lack of determinism across reruns. We illustrate that methodology by analyzing Matching Networks, Prototypical Networks and TADAM on the miniImagenet dataset.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/20/2022

Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis

Conjoint analysis is a popular experimental design used to measure multi...
research
05/09/2022

Methodology to Create Analysis-Naive Holdout Records as well as Train and Test Records for Machine Learning Analyses in Healthcare

It is common for researchers to holdout data from a study pool to be use...
research
11/06/2018

"I had a solid theory before but it's falling apart": Polarizing Effects of Algorithmic Transparency

The rise of machine learning has brought closer scrutiny to intelligent ...
research
11/29/2018

Correspondence Analysis of Government Expenditure Patterns

We analyze expenditure patterns of discretionary funds by Brazilian cong...
research
07/30/2021

Strategically using Applied Machine Learning for Accessibility Documentation in the Built Environment

There has been a considerable amount of research aimed at automating the...
research
08/31/2021

Double Machine Learning for Partially Linear Mixed-Effects Models with Repeated Measurements

Traditionally, spline or kernel approaches in combination with parametri...
research
01/14/2015

Quantifying Prosodic Variability in Middle English Alliterative Poetry

Interest in the mathematical structure of poetry dates back to at least ...

Please sign up or login with your details

Forgot password? Click here to reset