Recommending Training Set Sizes for Classification

02/16/2021
by   Phillip Koshute, et al.
0

Based on a comprehensive study of 20 established data sets, we recommend training set sizes for any classification data set. We obtain our recommendations by systematically withholding training data and developing models through five different classification methods for each resulting training set. Based on these results, we construct accuracy confidence intervals for each training set size and fit the lower bounds to inverse power low learning curves. We also estimate a sufficient training set size (STSS) for each data set based on established convergence criteria. We compare STSS to the data sets' characteristics; based on identified trends, we recommend training set sizes between 3000 and 30000 data points, according to a data set's number of classes and number of features. Because obtaining and preparing training data has non-negligible costs that are proportional to data set size, these results afford the potential opportunity for substantial savings for predictive modeling efforts.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/05/2012

Making Early Predictions of the Accuracy of Machine Learning Applications

The accuracy of machine learning systems is a widely studied research to...
research
04/05/2019

Simulation of virtual cohorts increases predictive accuracy of cognitive decline in MCI subjects

The ability to predict the progression of biomarkers, notably in NDD, is...
research
01/14/2019

Towards Testing of Deep Learning Systems with Training Set Reduction

Testing the implementation of deep learning systems and their training r...
research
02/27/2020

The Data Representativeness Criterion: Predicting the Performance of Supervised Classification Based on Data Set Similarity

In a broad range of fields it may be desirable to reuse a supervised cla...
research
01/05/2022

Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation

Common designs of model evaluation typically focus on monolingual settin...
research
07/04/2022

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Given a small training data set and a learning algorithm, how much more ...
research
11/11/2019

Rethinking Generalisation

In this paper, we present a new approach to computing the generalisation...

Please sign up or login with your details

Forgot password? Click here to reset