Small Data, Big Decisions: Model Selection in the Small-Data Regime

by   Jörg Bornschein, et al.

Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it. In contrast to most previous work, which typically considers the performance as a function of the model size, in this paper we empirically study the generalization performance as the size of the training set varies over multiple orders of magnitude. These systematic experiments lead to some interesting and potentially very useful observations; perhaps most notably that training on smaller subsets of the data can lead to more reliable model selection decisions whilst simultaneously enjoying smaller computational costs. Our experiments furthermore allow us to estimate Minimum Description Lengths for common datasets given modern neural network architectures, thereby paving the way for principled model selection taking into account Occams-razor.



page 1

page 2

page 3

page 4


Differential Description Length for Hyperparameter Selection in Machine Learning

This paper introduces a new method for model selection and more generall...

Assessing the Significance of Model Selection in Ecology

Model Selection is a key part of many ecological studies, with Akaike's ...

In Search of Lost Domain Generalization

The goal of domain generalization algorithms is to predict well on distr...

Model Selection in Bayesian Neural Networks via Horseshoe Priors

Bayesian Neural Networks (BNNs) have recently received increasing attent...

Towards a Theoretical Framework of Out-of-Distribution Generalization

Generalization to out-of-distribution (OOD) data, or domain generalizati...

A Model of One-Shot Generalization

We provide a theoretical framework to study a phenomenon that we call on...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.