Recommending Training Set Sizes for Classification

by   Phillip Koshute, et al.

Based on a comprehensive study of 20 established data sets, we recommend training set sizes for any classification data set. We obtain our recommendations by systematically withholding training data and developing models through five different classification methods for each resulting training set. Based on these results, we construct accuracy confidence intervals for each training set size and fit the lower bounds to inverse power low learning curves. We also estimate a sufficient training set size (STSS) for each data set based on established convergence criteria. We compare STSS to the data sets' characteristics; based on identified trends, we recommend training set sizes between 3000 and 30000 data points, according to a data set's number of classes and number of features. Because obtaining and preparing training data has non-negligible costs that are proportional to data set size, these results afford the potential opportunity for substantial savings for predictive modeling efforts.


page 1

page 2

page 3

page 4


Making Early Predictions of the Accuracy of Machine Learning Applications

The accuracy of machine learning systems is a widely studied research to...

Simulation of virtual cohorts increases predictive accuracy of cognitive decline in MCI subjects

The ability to predict the progression of biomarkers, notably in NDD, is...

Towards Testing of Deep Learning Systems with Training Set Reduction

Testing the implementation of deep learning systems and their training r...

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Given a small training data set and a learning algorithm, how much more ...

Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation

Common designs of model evaluation typically focus on monolingual settin...

The Data Representativeness Criterion: Predicting the Performance of Supervised Classification Based on Data Set Similarity

In a broad range of fields it may be desirable to reuse a supervised cla...

Understanding Isomorphism Bias in Graph Data Sets

In recent years there has been a rapid increase in classification method...