Log In Sign Up

On the Consistency of the Bootstrap Approach for Support Vector Machines and Related Kernel Based Methods

by   Andreas Christmann, et al.

It is shown that bootstrap approximations of support vector machines (SVMs) based on a general convex and smooth loss function and on a general kernel are consistent. This result is useful to approximate the unknown finite sample distribution of SVMs by the bootstrap approach.


page 1

page 2

page 3

page 4


Qualitative Robustness of Support Vector Machines

Support vector machines have attracted much attention in theoretical and...

On the stability of bootstrap estimators

It is shown that bootstrap approximations of an estimator which is based...

Support Vector Machines for Additive Models: Consistency and Robustness

Support vector machines (SVMs) are special kernel based methods and belo...

Approximate Stochastic Subgradient Estimation Training for Support Vector Machines

Subgradient algorithms for training support vector machines have been qu...

Function Estimation via Reconstruction

This paper introduces an interpolation-based method, called the reconstr...

Support Vector Regression, Smooth Splines, and Time Series Prediction

Prediction of dynamical time series with additive noise using support ve...

On the consistency of Multithreshold Entropy Linear Classifier

Multithreshold Entropy Linear Classifier (MELC) is a recent classifier i...

1 Introduction

Support vector machines and related kernel based methods can be considered as a hot topic in machine learning because they have good statistical and numerical properties under weak assumptions and have demonstrated their often good generalization properties in many applications, see e.g.

Vapnik1995 ; Vapnik1998 , ScSm2002 , and SteinwartChristmann2008a . To our best knowledge, the original SVM approach by BoserGuyonVapnik1992 was derived from the generalized portrait algorithm invented earlier by VapnikLerner1963 . Throughout the paper, the term SVM will be used in the broad sense, i.e. for a general convex loss function and a general kernel.

SVMs based on many standard kernels as for example the Gaussian RBF kernel are nonparametric methods. The finite sample distribution of many nonparametric methods is unfortunately unknown because the distribution from which the data were generated is usually completely unknown and because there are often only asymptotical results describing the consistency or the rate of convergence of such methods known so far. Furthermore, there is in general no uniform rate of convergence for such nonparametric methods due to the famous no-free-lunch theorem, see Devroye1982 and DevroyeGyoerfiLugosi1996

. Informally speaking, the no-free-lunch theorem states that, for sufficiently malign distributions, the average risk of any statistical (classification) method may tend arbitrarily slowly to zero. Theses facts are true for SVMs. SVMs are known to be universally consistent and fast rates of convergence are known for broad


of all probability distributions. The asymptotic normality of SVMs was shown recently by

Hable2012 under certain conditions.

Here, we apply a different approach to SVMs, namely Efron’s bootstrap. The goal of this paper is to show that bootstrap approximations of SVMs which are based on a general convex and smooth loss function and a general smooth kernel are consistent under mild assumptions; more precisely, convergence in outer probability is shown. This result is useful to draw statistical decisions based on SVMs, e.g. confidence intervals, tolerance intervals and so on.

We mention that both the sequence of SVMs and the sequence of their corresponding risks are qualitatively robust under mild assumptions, see ChristmannSalibianBarreraVanAelst2013 . Hence, Efron’s bootstap approach turns out to be quite successful for SVMs from several aspects.

The rest of the paper has the following structure. Section 2 gives a brief introduction into SVMs. Section 3 gives the result. The last section contains the proof and related results.

2 Support Vector Machines

Current statistical applications are characterized by a wealth of large and high-dimensional data sets. In classification and in regression problems there is a variable of main interest, often called “output values” or “response”, and a number of potential explanatory variables, which are often called “input values”. These input values are used to model the observed output values or to predict future output values. The observations consist of

pairs , …, , which will be assumed to be independent realizations of a random pair . We are interested in minimizing the risk or to obtain a function such that is a good predictor for the response , if is observed. The prediction should be made in an automatic way. We refer to this process of determining a prediction method as “statistical machine learning”, see e.g. Vapnik1995 ; Vapnik1998 ; ScSm2002 ; CuckerZhou2007 ; SmaleZhou2007 . Here, by “good predictor” we mean that minimizes the expected loss, i.e. the risk,


denotes the unknown joint distribution of the random pair

and is a fixed loss function. As a simple example, the least squares loss yields the optimal predictor , . Because is unknown, we can neither compute nor minimize the risk directly.

Support vector machines, see VapnikLerner1963 , BoserGuyonVapnik1992 , Vapnik1995 ; Vapnik1998 , provide a highly versatile framework to perform statistical machine learning in a wide variety of setups. The minimization of regularized empirical risks over reproducing kernel Hilbert spaces was already considered e.g. by PoggioGirosi1990 . Given a kernel we consider predictors , where denotes the corresponding reproducing kernel Hilbert space of functions from to . The space includes, for example, all functions of the form where are arbitrary elements in and , . To avoid overfitting, a support vector machine is defined as the solution of a regularized risk minimization problem. More precisely,


where is the regularization parameter. For a sample

the corresponding estimated function is given by


where denotes the empirical distribution based on (see (2.3) below). Note that the optimization problem (2.2) corresponds to (2.1) when using instead of .

Efficient algorithms to compute exist for a number of different loss functions. However, there are often good reasons to consider other convex loss functions, e.g. the hinge loss for binary classification purposes or the -insensive loss for regression purposes, where . As these loss functions are not differentiable, the logistic loss functions and and Huber-type loss functions are also used in practice. These loss functions can be considered as smoothed versions of the previous two loss functions.

An important component of statistical analyses concerns quantifying and incorporating uncertainty (e.g. sampling variability) in the reported estimates. For example, one may want to include confidence bounds along the individual predicted values obtained from (2.2). Unfortunately, the sampling distribution of the estimated function is unknown. Recently, Hable2012 derived the asymptotic distribution of SVMs under some mild conditions. Asymptotic confidence intervals based on those general results are always symmetric.

Here, we are interested in approximating the finite sample distribution of SVMs by Efron’s bootstrap approach, because confidence intervals based on the bootstrap approach can be asymmetric. The bootstrap Efron1979 provides an alternative way to estimate the sampling distribution of a wide variety of estimators. To fix ideas, consider a functional , where is a set of probability measures and denotes a metric space. Many estimators can be included in this framework. Simple examples include the sample mean (with functional ) and M-estimators (with functional defined implicitly as the solution to the equation ). Let be the Borel -algebra on and denote the set of all Borel probability measures on by . Then, it follows that (2.1) defines an operator

i.e. the support vector machine. Moreover, the estimator in (2.2) satisfies



is the empirical distribution based on the sample and denotes the Dirac measure at the point .

More generally, let ,

, be independent and identically distributed (i.i.d.) random variables with distribution

, and let

be the corresponding estimator, where

Denote the distribution of by . If was known to us, we could estimate this sampling distribution by drawing a large number of random samples from and evaluating our estimator on them. The basic idea of Efron’s bootstrap approach is to replace the unknown distribution by an estimate . Here we will consider the natural non-parametric estimator given by the sample empirical distribution . In other words, we estimate the distribution of our estimator of interest by its sampling distribution when the data are generated by . In symbols, the bootstrap proposes to use

Since this distribution is generally unknown, in practice one uses Monte Carlo simulation to estimate it by repeatedly evaluating the estimator on samples drawn from . Note that drawing a sample from means that observations are drawn with replacement from the original observations , …, .

3 Consistency of Bootstrap SVMs

In this section it will be shown under appropriate assumptions that the weak consistency of bootstrap estimators carries over to the Hadamard-differentiable SVM functional in the sense that the sequence of “conditional random laws” (given ) of is asymptotically consistent in probability for estimating the laws of the random elements . In other words, if is large, the ”random distribution”


based on bootstrapping an SVM can be considered as a valid approximation of the unknown finite sample distribution

Assumption 3.1

Let be closed and bounded and let be closed. Assume that is the restriction of an -times continuously differentiable kernel such that and . Let be the RKHS of and let be a probability distribution on . Let be a convex, -square-integrable Nemitski loss function of order such that the partial derivatives

exist for every . Assume that the maps

are continuous. Furthermore, assume that for every , there is a and a constant such that, for every ,


The conditions on the kernel in Assumption 3.1 are satisfied for many common kernels, e.g., Gaussian RBF kernel, exponential kernel, polynomial kernel, and linear kernel, but also Wendland kernels based on certain univariate polynomials of degree for such that , see Wendland2005 .

The conditions on the loss function in Assumption 3.1 are satisfied, e.g., for the logistic loss for classification or for regression, however the popular non-smooth loss functions hinge, -insensitive, and pinball are not covered. However, (Hable2012, , Remark 3.5) described an analytical method to approximate such non-smooth loss functions up to an arbitrarily good precision by a convex -square integrable Nemitski loss function of order .

We can now state our result on the consistency of the bootstrap approach for SVMs.

Theorem 3.2

Let Assumption 3.1 be satisfied. Let . Then


converge in outer probability, where is a tight Borel-measurable Gaussian process, is a continuous linear operator with




is a continuous linear operator which is invertible.

For details on , , and we refer to Lemma 1, Theorem 4.4, and Lemma 2.

4 Proofs

4.1 Tools for the proof of Theorem 3.2

We will need two general results on bootstrap methods proven in VandervaartWeller1996 and adopt their notation, see (VandervaartWeller1996, , Chapters 3.6 and 3.9). Let be the empirical measure of an i.i.d. sample from a probability distribution . The empirical process is the signed measure

Given the sample values, let be an i.i.d. sample from . The bootstrap empirical distribution is the empirical measure , and the bootstrap empirical process is

where is the number of times that is “redrawn” from the original sample , is stochastically independent of and multinomially distributed with parameters and probabilities . If outer expectations are computed, stochastic independence is understood in terms of a product probability space. Let be the coordinate projections on the first coordinates of the product space and let the multinomial vectors depend on the last factor only, see (VandervaartWeller1996, , p. 345f).

The following theorem shows (conditional) weak convergence for the empirical bootstrap, where the symbol denotes the weak convergence of finite measures. We will need only the equivalence between and from this theorem and list part only for the sake of completeness.

Theorem 4.1 ((VandervaartWeller1996, , Thm. 3.6.2, p. 347))

Let be a class of measurable functions with finite envelope function. Define . The following statements are equivalent:

  1. is Donsker and ;

  2. converges outer almost surely to zero and the sequence converges almost surely to zero for every .

  3. converges outer almost surely to zero and the sequence converges almost surely to zero for every .

Here the asterisks denote the measurable cover functions with respect to , , and jointly.

Consider sequences of random elements and in a normed space such that the sequence converges unconditionally and the sequence converges conditionally on in distribution to a tight random element . A precise formulation of the second assumption is


in outer probability, with ranging over the bounded Lipschitz functions, see (VandervaartWeller1996, , p. 378, Formula (3.9.9)). The next theorem shows that under appropriate assumptions, weak consistency of the bootstrap estimators carries over to any Hadamard-differentiable functional in the sense that the sequence of “conditional random laws” (given ) of is asymptotically consistent in probability for estimating the laws of the random elements , see (VandervaartWeller1996, , p.378).

Theorem 4.2 ((VandervaartWeller1996, , Thm. 3.9.11, p. 378))

(Delta-method for bootstrap in probability) Let and be normed spaces. Let be Hadamard-differentiable at tangentially to a subspace . Let and be maps as indicated previously with values in such that and that (4.11)-(4.12) holds in outer probability, where is separable and takes its values in . Then


holds in outer probability.

As was pointed out by (VandervaartWeller1996, , p. 378), consistency in probability appears to be sufficient for (many) statistical purposes and the theorem above shows this is retained under Hadamard differentiability at the single distribution .

We now list some results from Hable2012 , which will also be essential for the proof of Theorem 3.2.

Theorem 4.3 ((Hable2012, , Theorem 3.1))

Let Assumption 3.1 be satisfied. Then, for every regularizing parameter , there is a tight, Borel-measurable Gaussian process , , such that


for every Borel-measurable sequence of random regularization parameters with in probability. The Gaussian process is zero-mean; i.e., for every .

Lemma 1 ((Hable2012, , Lemma A.5))

For every defined later in (4.25),


is a continuous linear operator which is invertible.

Theorem 4.4 ((Hable2012, , Theorem A.8))

For every which fulfills , the map , , is Hadamard-differentiable in tangentially to the closed linear span . The derivative in is a continuous linear operator such that

Lemma 2 ((Hable2012, , Lemma A.9))

For every data set , let denote the element of which corresponds to the empirical measure . That is, for every . Then


where is a tight Borel-measurable Gaussian process such that for every .

4.2 Proof of Theorem 3.2

The proof relies on the application of Theorem 4.2. Hence, we have to show the following steps:

  1. The empirical process weakly converges to a separable Gaussian process .

  2. SVMs are based on a map which is Hadamard differentiable at tangentially to some appropriate subspace.

  3. The assumptions (4.11)-(4.12) of Theorem 4.2 are satisfied. For this purpose we will use Theorem 4.1. Actually, we will show that part (i) of Theorem 4.1 is satisfied which gives the equivalence to part (iii), from which we conclude that (4.11)-(4.12) hold true. For the proof that part (i) of Theorem 4.1 is satisfied, i.e., that a suitable set is a -Donsker class and that , we use several facts recently shown by Hable2012 .

  4. We put all parts together and apply Theorem 4.2.

Step 1.  To apply Theorem 4.2, we first have to specify the considered spaces , , , and the map . As in Hable2012 we use the following notations. Because is a -square-integrable Nemitski loss function of order , there is a function such that








is the set of all indicator functions ,




Now let be the set of all bounded functions with norm . Define




the closed linear span of in . That is, is a subset of whose elements correspond to finite measures. Hence probability measures are covered as special cases. The elements of can be interpreted as some kind of generalized distributions functions, because . The assumptions on and imply that , is a well-defined element of . For every , let denote the corresponding finite measure on such that for all . Note that the map is well-defined, because by definition of , uniquely exists for every .

With these notations, we will apply Theorem 4.2 for


At first glance this definition of seems to be somewhat technical. However, this will allow us to use a functional delta method for bootstrap estimators of SVMs with regularization parameter .

Lemma 2 guarantees that the empirical process weakly converges to a tight Borel-measurable Gaussian process.

Since a -compact set in a metric space is separable, separability of a random variable is slightly weaker than tightness, see (VandervaartWeller1996, , p. 17). Therefore, in our Theorem 3.2 is indeed separable.

Step 2.  Theorem 4.4 showed that the map indeed satisfies the necessary Hadamard-differentiability in the point .

Step 3.  We know that is a -Donsker class, see Lemma 2. Hence, an immediate consequence from (VandervaartWeller1996, , Theorem 3.6.1, p. 347) is, that


converges in outer probability to zero and is asymptotically measurable.

However, we will prove a somewhat stronger result, namely that is a -Donsker class and , which is part (i) of Theorem 4.1, and then part (iii) of Theorem 4.1 yields, that the term in (4.28) converges even outer almost surely to zero and the sequence


converges almost surely to zero for every .

Because is a -Donsker class, it remains to show that . Due to


and , we obtain the inequality


We will show that each of the three summands on the right hand side of the last inequality is finite. If , then equals the indicator function for some . Hence, and the summand for is finite. If , then because is by assumption a -square-integrable Nemitski loss function of order . Hence the summand for is finite, too. Let us now consider the case that . By definition of , for every there exist such that , , and , where we used the notation for all . Using for every , we obtain


Define the constant with given by (4.20). Hence, for all ,


Hence we get

because and by Assumption 3.1. With the same arguments we obtain, for every ,

because and by Assumption 3.1. Hence,

Therefore, the sum on the right hand side in (4.31) is finite and thus the assumption is satisfied. This yields by part (iii) of Theorem 4.1 that converges outer almost surely to zero and the sequence


converges almost surely to zero for every , where the asterisks denote the measurable cover functions with respect to and jointly.

Step 4.  Due to Step 3, the assumption (4.11) of Theorem 4.2 is satisfied. We now show that additionally (4.12) is satisfied, i.e., that the term in (4.34) converges to zero in outer probability. In general, one can not conclude that almost sure convergence implies convergence in outer probability, see (VandervaartWeller1996, , p. 52). We know that the term in (4.34) converges almost surely to zero for every , where the asterisks denote the measurable cover functions with respect to and jointly. Hence, for every , the cover functions to be considered in (4.34) are measurable. Additionally, the multinomially distributed random variable is stochastically independent of in the bootstrap, where independence is understood in terms of a product probability space, see (VandervaartWeller1996, , p. 346) for details. Therefore, an application of the Fubini-Tonelli theorem, see e.g., (DenkowskiEtAl2003, , p. 174, Thm. 2.4.10), yields that the inner integral considered by Fubini-Tonelli is measurable for every and every . Recall that almost sure convergence of measurable functions implies convergence in probability which is equivalent with convergence in outer probability for measurable functions. Hence we have convergence in outer probability in (4.34). Therefore, all assumptions of Theorem 4.2 are satisfied and the assertion of our theorem follows.


  • [1] B. E. Boser, I. Guyon, and V. Vapnik.

    A training algorithm for optimal margin classifiers.

    In Computational Learning Theory, pages 144–152, Madison, WI, 1992. ACM.
  • [2] A. Christmann, M. Salibían-Barrera, and S. Van Aelst. Qualitative Robustness of Bootstrap Approximations for Kernel Based Methods, chapter 16 in C. Becker, R. Fried, S. Kuhnt (Eds.). “Robustness and Complex Data Structures” (Preprint available on Springer, Heidelberg, 2013.
  • [3] F. Cucker and D.X. Zhou. Learning Theory: An Approximation Theory Viewpoint. Cambridge University Press, Cambridge, 2007.
  • [4] Z. Denkowski, S. Migórski, and N.S. Papageorgiou. An introduction to nonlinear analysis: Theory. Kluwer Academic Publishers, Boston, 2003.
  • [5] L. Devroye. Any discrimination rule can have an arbitrarily bad probability of error for finite sample size. IEEE Trans. Pattern Anal. Mach. Intell., 4:154–157, 1982.
  • [6] L. Devroye, L. Györfi, and G. Lugosi.

    A Probabilistic Theory of Pattern Recognition

    Springer, New York, 1996.
  • [7] B. Efron. Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7:1–26, 1979.
  • [8] R. Hable. Asymptotic normality of support vector machine variants and other regularized kernel methods.

    Journal of Multivariate Analysis

    , 106:92–117, 2012.
  • [9] T. Poggio and F. Girosi. Networks for approximation and learning. Proc. IEEE, 78:1481–1497, 1990.
  • [10] B. Schölkopf and A. J. Smola. Learning with Kernels. Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, 2002.
  • [11] S. Smale and D.-X. Zhou. Learning theory estimates via integral operators and their approximations. Constr. Approx., 26:153–172, 2007.
  • [12] I. Steinwart and A. Christmann. Support Vector Machines. Springer, New York, 2008.
  • [13] A. W. van der Vaart and J. A. Wellner. Weak convergence and empirical processes. Springer, New York, 1996.
  • [14] V. N. Vapnik.

    The Nature of Statistical Learning Theory

    Springer, New York, 1995.
  • [15] V. N. Vapnik. Statistical Learning Theory. John Wiley & Sons, New York, 1998.
  • [16] V. N. Vapnik and A. Lerner. Pattern recognition using generalized portrait method. Autom. Remote Control, 24:774–780, 1963.
  • [17] H. Wendland. Scattered Data Approximation. Cambridge University Press, Cambridge, 2005.