DeepAI AI Chat
Log In Sign Up

How little data do we need for patient-level prediction?

by   Luis H. John, et al.

Objective: Provide guidance on sample size considerations for developing predictive models by empirically establishing the adequate sample size, which balances the competing objectives of improving model performance and reducing model complexity as well as computational requirements. Materials and Methods: We empirically assess the effect of sample size on prediction performance and model complexity by generating learning curves for 81 prediction problems in three large observational health databases, requiring training of 17,248 prediction models. The adequate sample size was defined as the sample size for which the performance of a model equalled the maximum model performance minus a small threshold value. Results: The adequate sample size achieves a median reduction of the number of observations between 9.5 0.02. The median reduction of the number of predictors in the models at the adequate sample size varied between 8.6 Discussion: Based on our results a conservative, yet significant, reduction in sample size and model complexity can be estimated for future prediction work. Though, if a researcher is willing to generate a learning curve a much larger reduction of the model complexity may be possible as suggested by a large outcome-dependent variability. Conclusion: Our results suggest that in most cases only a fraction of the available data was sufficient to produce a model close to the performance of one developed on the full data set, but with a substantially reduced model complexity.


page 30

page 31

page 32

page 33

page 34

page 40

page 41

page 42


Second Order Expansions for Sample Median with Random Sample Size

In practice, we often encounter situations where a sample size is not de...

Sample size determination via learning-type curves

This paper is concerned with sample size determination methodology for p...

The need for adequate sampling in a well-functioning market surveillance system

Adequate sampling is essential for the well-functioning of a market surv...

Unravelling Small Sample Size Problems in the Deep Learning World

The growth and success of deep learning approaches can be attributed to ...

Fitting Semiparametric Cumulative Probability Models for Big Data

Cumulative probability models (CPMs) are a robust alternative to linear ...

Sample Size Planning for Classification Models

In biospectroscopy, suitably annotated and statistically independent sam...

Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

This work presents a statistically principled method for estimating the ...