( β, ϖ)-stability for cross-validation and the choice of the number of folds
share
In this paper, we introduce a new concept of stability for cross-validation, called the ( β, ϖ)-stability, and use it as a new perspective to build the general theory for cross-validation. The ( β, ϖ)-stability mathematically connects the generalization ability and the stability of the cross-validated model via the Rademacher complexity. Our result reveals mathematically the effect of cross-validation from two sides: on one hand, cross-validation picks the model with the best empirical generalization ability by validating all the alternatives on test sets; on the other hand, cross-validation may compromise the stability of the model selection by causing subsampling error. Moreover, the difference between training and test errors in qth round, sometimes referred to as the generalization error, might be autocorrelated on q. Guided by the ideas above, the ( β, ϖ)-stability help us derivd a new class of Rademacher bounds, referred to as the one-round/convoluted Rademacher bounds, for the stability of cross-validation in both the i.i.d. and non-i.i.d. cases. For both light-tail and heavy-tail losses, the new bounds quantify the stability of the one-round/average test error of the cross-validated model in terms of its one-round/average training error, the sample sizes n, number of folds K, the tail property of the loss (encoded as Orlicz-Ψ_ν norms) and the Rademacher complexity of the model class Λ. The new class of bounds not only quantitatively reveals the stability of the generalization ability of the cross-validated model, it also shows empirically the optimal choice for number of folds K, at which the upper bound of the one-round/average test error is lowest, or, to put it in another way, where the test error is most stable.
READ FULL TEXT
