The Shape of Learning Curves: a Review

03/19/2021
by   Tom Viering, et al.
0

Learning curves provide insight into the dependence of a learner's generalization performance on the training set size. This important tool can be used for model selection, to predict the effect of more training data, and to reduce the computational complexity of model training and hyperparameter tuning. This review recounts the origins of the term, provides a formal definition of the learning curve, and briefly covers basics such as its estimation. Our main contribution is a comprehensive overview of the literature regarding the shape of learning curves. We discuss empirical and theoretical evidence that supports well-behaved curves that often have the shape of a power law or an exponential. We consider the learning curves of Gaussian processes, the complex shapes they can display, and the factors influencing them. We draw specific attention to examples of learning curves that are ill-behaved, showing worse learning performance with more training data. To wrap up, we point out various open problems that warrant deeper empirical and theoretical investigation. All in all, our review underscores that learning curves are surprisingly diverse and no universal model can be identified.

READ FULL TEXT
research
11/25/2022

A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance

Plotting a learner's generalization performance against the training set...
research
07/11/2019

Minimizers of the Empirical Risk and Risk Monotonicity

Plotting a learner's average performance against the number of training ...
research
01/28/2022

Learning Curves for Decision Making in Supervised Machine Learning – A Survey

Learning curves are a concept from social sciences that has been adopted...
research
04/07/2020

A Brief Prehistory of Double Descent

In their thought-provoking paper [1], Belkin et al. illustrate and discu...
research
11/25/2020

Learning Curves for Drug Response Prediction in Cancer Cell Lines

Motivated by the size of cell line drug sensitivity data, researchers ha...
research
01/30/2020

Analytic Study of Double Descent in Binary Classification: The Impact of Loss

Extensive empirical evidence reveals that, for a wide range of different...
research
09/03/2018

Proper likelihood ratio based ROC curves for general binary classification problems

Everybody writes that ROC curves, a very common tool in binary classific...

Please sign up or login with your details

Forgot password? Click here to reset