Support vector machine (SVM) is by definition the Tikhonov regularization associated with some loss function over a reproducing kernel Hilbert space (RKHS). Due to its clear statistical properties (Zhang, 2004; Blanchard et al., 2008) and fast learning rates (Steinwart and Scovel, 2007; Tong, 2016), SVM has triggered enormous research activities in the past twenty years. Theoretical assessments for the feasibility of SVM have been widely studied, to just name a few, (Chen et al., 2004; Wu and Zhou, 2005; Wang, 2005; Zhou and Jetter, 2006; Cucker and Zhou, 2007; Wu et al., 2007; Tong et al., 2008; Steinwart and Christmann, 2008).
As shown in (Steinwart, 2002), selecting a suitable kernel facilitates the use of SVM, both in theoretical analysis and practical applications. Gaussian kernel is one of the most important kernels in practice, where the width of the Gaussian kernel reflects the frequency information for a specified learning problem (Keerthi and Lin, 2003). Structures as well as explicit representations of the inner products and norms of Gaussian RKHS have been studied in (Steinwart et al., 2006; Minh, 2010). Furthermore, tight bounds of various covering numbers for Gaussian RKHS were provided in (Zhou, 2002, 2003; Steinwart and Scovel, 2007; Kühn, 2011). Based on these developed bounds, fast learning rates of SVM with Gaussian kernel were derived in (Ying and Zhou, 2007; Steinwart and Scovel, 2007; Ye and Zhou, 2008; Xiang and Zhou, 2009; Xiang, 2011, 2012; Hu, 2011; Eberts and Steinwart, 2013; Lin et al., 2014, 2015). As a typical example, (Steinwart and Scovel, 2007) proved that there exist non-trivial distributions such that the learning rate of SVM classification with Gaussian kernel and hinge loss can reach an order of , where is the number of samples. Similar results were established for SVM with quadratic loss in (Xiang and Zhou, 2009).
This paper aims at refined analysis for SVM classification with convex loss and Gaussian kernel. Our first purpose is to derive almost optimal learning rates for SVM classification. Our result shows that if the regression function (see Section 2 for definition) is -smooth, then SVM with Gaussian kernel and certain loss functions, such as the quadratic loss and truncated quadratic loss, can reach a learning rate of order with arbitrarily small positive number . The learning rate was proved to be optimal in the minimax sense in (Yang, 1999) for classification with these loss functions when the regression function is smooth.
Since the rate is always slower than , our second purpose is to deduce fast learning rates of SVM with Gaussian kernel under additional assumptions on the regression function. In particular, we find that if the regression function is infinitely differentiable and the Tsybakov noise exponent (Tsybakov, 2004) tends to infinity, then SVM with Gaussian kernel and loss functions including the hinge loss, quadratic loss, and truncated quadratic loss can achieve an order of . This implies that there exist non-trivial distributions such that learning rates of SVM with Gaussian kernel can reach the order of , which extends the results in (Steinwart and Scovel, 2007; Xiang and Zhou, 2009) for the hinge loss and quadratic loss to a general case.
The rest of paper is organized as follows. Section 2 presents some definitions and introduces the algorithm studied in this paper. Section 3 provides the main results. Section 4 compares our results with some related work and gives some further discussions. Section 5 establishes two oracle inequalities for SVM with convex loss. Section 6 gives the proofs of the main results.
2 Classification with Gaussian Kernel and Convex Loss
. Binary classification algorithms produce a classifier, whose generalization ability is measured by the misclassification error
where is the marginal distribution of and
is the conditional probability at. The Bayes rule
minimizes the misclassification error, where is the Bayes decision function. Since is independent of the classifier , the performance of can be measured by the excess misclassification error .
Given a loss function , denote by the generalization error with respect to and by
the regression function minimizing . If is differentiable, it is easy to check that
We are concerned with the hinge loss and the following twice smooth classifying loss.
We say that is a classifying loss (function), if it is convex, differentiable at with , and the smallest zero of is . We say that is a twice smooth classifying loss, if in addition, it is differentialable, and its derivative is continuous and satisfies
and its modulus of convexity satisfies
The classifying loss was defined in (Xiang and Zhou, 2009) and the modulus of convexity together with condition (4) was given in (Bartlett et al., 2006). It is easy to check that the quadratic loss and truncated quadratic loss (or 2-norm hinge loss) are twice smooth classifying loss. It should be mentioned that the twice smooth classifying loss is different from the loss of quadratic type defined in (Koltchinaskii and Yuan, 2010), since the classifying loss requiring a zero point of , deports the well known logistic loss, a typical loss of quadratic type, while the twice differentiable property of the loss of quadratic type deports the truncated quadratic loss. As concrete examples of our analysis, we are specifically interested in loss functions presented in Table 1. All of them are frequently used in practical applications (Bartlett et al., 2006). The regression functions of other twice smooth classifying loss can be deduced from (2). Since the subgradient of at is not unique, the regression function for is not unique too. In Table 1, for not close to , we set but allow when , when and when .
|Loss function||Mathematical representation||Regression function|
be the Gaussian kernel, where is the width of and denotes the Euclidean norm. Denote by the RKHS associated with endowed with the inner product and norm . We consider learning rates of the following algorithm
where is a regularization parameter.
3 Main Results
Our error analysis is built upon a smoothness assumption on the regression function, which requires the following definition.
Let for some and . A function is said to be -smooth if for every , the partial derivatives exist and satisfy
Denote by the set of all -smooth functions.
To derive the learning rate, we need the following assumption.
for some and .
Assumption 1 describes the smoothness and boundedness of the regression function. If is the quadratic or truncated quadratic loss, then the smoothness of the regression function is equivalent to the smoothness of the Bayes decision function . If is the hinge loss, is not unique. Assumption 1 means that there is an and implies that and have a strictly positive distance, which is a bit strict. Hence, for SVM with hinge loss, a preferable assumption is a geometric noise assumption introduced in (Steinwart and Scovel, 2007, Definition 2.3) (see also (Steinwart and Christmann, 2008, Definition 8.15)). Presenting learning results for SVM with hinge loss under Assumption 1 in this paper is for the sake of completeness.
Based on Assumption 1, we present our first main result.
Let , be either the hinge loss or a twice smooth classifying loss. If Assumption 1 holds, and , then for arbitrary , with confidence at least , there holds
where is a positive constant independent of or .
With the help of the above confidence-based error estimate, we can derive the following learning rate in expectation.
Let be either the hinge loss or a twice smooth classifying loss. If Assumption 1 holds, and , then for arbitrary , there holds
where is specified in Theorem 1.
Corollary 1 gives an upper bound for algorithm (5) with the hinge loss and twice smooth classifying loss under Assumption 1. However, it is difficult to judge whether the bound is tight for all these loss functions. We obtain in the following corollary that at least for certain specified loss functions, the error estimate in (7) is almost optimal.
Let be either the quadratic loss or truncated quadratic loss. If , , then for arbitrary , there holds
where is a constant independent of , and was specified in Theorem 1.
It should be mentioned that depends on and the supremum on is equivalent to maximizing on some set of functions. Although the learning rate derived in (8) is almost optimal, it is always slower than . We then aim at deriving fast learning rates for algorithm (5) by imposing additional conditions on the distribution . For this purpose, we need the following Tsybakov noise condition (Tsybakov, 2004).
Let . We say that satisfies the Tsybakov noise condition with exponent if there exists a constant such that
To derive the fast learning rate, we need the following assumption.
satisfies the Tsybakov noise condition with exponent .
It can be found in (Tsybakov, 2004) that Assumption 2 measures the size of the set of points that are corrupted with high noise in the labeling process, and always holds for with . It has been adopted in (Steinwart and Scovel, 2007; Xiang and Zhou, 2009; Xiang, 2011; Tong, 2016) to deduce fast learning rates for SVM with various loss functions. Noting that Assumption 1 reflects the smoothness of while Assumption 2 measures the level of critical noise, these two assumptions are compatible in some sense. A simple example is that when
is the uniform distribution on, is the quadratic loss and , and satisfy Assumption 1 with and some . Furthermore, plugging into (9), Assumption 2 holds with and . The following two theorems show the improved learning rates under Assumptions 1 and 2.
When , Theorems 2 and 3 coincide with Theorem 1. If , which implies that the approximation error approaches to , then the learning rates derived in Theorems 2 and 3 are . These rates coincide with the optimal learning rates for certain classifiers based on empirical risk minimization in (Tsybakov, 2004) up to an arbitrarily small positive number , and are the same as those presented in (Steinwart and Christmann, 2008, Chapter 8) for the hinge loss. Based on Theorems 2 and 3, we can deduce the following corollary, showing that classification with Gaussian kernel for a large number of loss functions can reach the rate for nontrivial distributions.
4 Related Work and Discussion
SVM with Gaussian kernel and convex loss is a state-of-the-art learning strategy for tackling regression and classification problems. For the regression purpose, almost optimal learning rates of SVM with Gaussian kernel and quadratic loss were derived in (Eberts and Steinwart, 2013). From regression to classification, comparison inequalities play crucial roles in analysis. Given a classifier and some convex loss function , the comparison inequality in (Chen et al., 2004) showed that the excess misclassification error can be bounded by means of the generalization error :
Furthermore, for , (Zhang, 2004) showed that
For other loss functions, learning rates of classification with Gaussian kernel were deduced in (Steinwart and Scovel, 2007; Xiang and Zhou, 2009; Xiang, 2011, 2012). In particular, (Steinwart and Scovel, 2007) proved that there exist non-trivial distributions (geometric noise assumptions for the distribution and Tsybakov noise conditions) such that learning rates of SVM with Gaussian kernel and hinge loss can reach an order of . Using the refined technique in approximation theory, (Xiang and Zhou, 2009) also constructed some distributions (smoothness assumptions for the regression function and Tsybakov noise conditions) such that learning rates of SVM with Gaussian kernel and quadratic loss can reach an order of . Moreover, (Xiang and Zhou, 2009) deduced learning rates for SVM with Gaussian kernel and classifying loss, including the norm hinge loss with and exponential hinge loss under some smoothness assumption similar to Assumption 1. When the loss function is twice differentiable, (Xiang, 2011) improved (Xiang and Zhou, 2009)’s results in terms of deriving fast learning rates of SVM under additional Tsybakov noise conditions. The main tool is the comparison inequality under Assumption 2 (Bartlett et al., 2006; Xiang, 2011) (see also (Steinwart and Christmann, 2008, Theorem 8.29)), saying that for arbitrary measurable function , there holds
where is a constant depending only on . Since the definition of the classifying loss in (Xiang and Zhou, 2009) deports the logistic loss and exponential loss, (Xiang, 2012) derived learning rates for SVM with some loss functions without the smallest zero restriction in the classifying loss. Under this circumstance, learning rates for SVM classification with Gaussian kernel and logistic loss were derived in (Xiang, 2012).
Under Assumption 1, we derive almost optimal learning rates for SVM with quadratic loss and truncated quadratic loss. The derived learning rate in (6) is better than the rates in (Xiang and Zhou, 2009, Theorem 1) with , while is the same as that rate derived in (Eberts and Steinwart, 2013) for the quadratic loss. Moreover, for the hinge loss, our result in (6) is better than that in (Xiang and Zhou, 2009, Theorem 4). Furthermore, Corollary 3 shows that for some non-trivial distributions (smoothness assumptions for the regression function and Tsybakov noise conditions), SVM with Gaussian kernel and hinge loss or twice smooth classifying loss can reach the learning rate of order with an arbitrarily small positive number . Our results extend the results in (Steinwart and Scovel, 2007) (for hinge loss) and (Xiang and Zhou, 2009) (for quadratic loss) to a general case. For another widely used kernel, the polynomial kernel with , learning rates for SVM with convex loss functions were deduced in (Zhou and Jetter, 2006; Tong et al., 2008). The detailed comparisons between our paper and (Xiang and Zhou, 2009) (XZ2009), (Eberts and Steinwart, 2013) (ES2013), (Tong et al., 2008) (T2008) are summarized in Table 2 and Table 3.
Besides the smoothness assumption on the regression function, (Steinwart and Scovel, 2007) proposed a geometric noise assumption with exponent (Steinwart and Scovel, 2007, Definition 2.3) to describe the learning rates for SVM. Based on that assumption and Assumption 2 in this paper, a learning rate of order was derived for SVM with Gaussian kernel and hinge loss. Under the same conditions as (Steinwart and Scovel, 2007), (Tong, 2016) derived a learning rate of order for SVM with polynomial kernels and hinge loss. As mentioned in the previous section, Assumption 1 for the hinge loss implies a strictly positive distance between and for arbitrary , which implies the geometric noise assumption with . Thus a learning rate of order can be derived for arbitrary . Under this circumstance, the smoothness index fails to describe the a-priori knowledge for the classification problems and we recommend to use the geometric noise assumption in (Steinwart and Scovel, 2007, Definition 2.3) or (Steinwart and Christmann, 2008, Definition 8.15) to quantify the a-priori information. The reason of introducing Assumption 1 to analyze the learning rate for SVM with hinge loss is for the sake of completeness and uniformity for analysis.
In this paper, we study the learning performance of SVM with Gaussian kernel and convex loss. The main tools are two oracle inequalities developed in the next section. Such two oracle inequalities are different from the standard result in (Steinwart and Christmann, 2008, Theorem 7.23) that is based on a very genral oracle inequality established in (Steinwart and Christmann, 2008, Theorem 7.20). To be detailed, (Steinwart and Christmann, 2008, Theorem 7.23) requires a polynomial decaying assumption on the (weaker) covering number of the RKHS but does not need the compactness of the input space or the continuity of the kernel, while our analysis needs Assumption 3 in Section 5, compactness of and continuity of . It should be mentioned that Assumption 3 contains the logarithmic decaying for the covering number, which requires some non-trivial additional work. We believe that by using the established oracle inequalities and approximation results in (Zhou and Jetter, 2006; Tong et al., 2008), similar error analysis for the polynomial kernel can be derived. As far as the Gaussian kernel is concerned, our results might be derived from the approximation error analysis in this paper and (Steinwart and Christmann, 2008, Theorem 7.23) with slight changes, by using the twice smoothness property (4) of the loss functions to verify conditions of (Steinwart and Christmann, 2008, Theorem 7.23). It would be interesting to derive learning rates for classification with online learning and Gaussian kernel (Hu, 2011) and classification with Gaussian kernel and convex loss when is a lower dimensional manifold (Ye and Zhou, 2008) by utilizing the approaches in this paper.
5 Oracle Inequalities for SVM with Convex Loss
In this section, we present two oracle inequalities for SVM with convex loss and Mercer kernels. Denote by the space of square integrable functions endowed with norm . Let be the RKHS associated with a Mercer kernel endowed with norm . Define
where is a regularization parameter. Our oracle inequalities are built upon the following Assumption 3.
where , is a decreasing and continuous function, is the ball in with some and denotes the covering number of (Xiang and Zhou, 2009).
Assumption 3 depicts the capacity of RKHS. It holds for RKHS with Gaussian kernel (Steinwart and Scovel, 2007) with for arbitrary , and for RKHS with polynomial kernel (Zhou and Jetter, 2006) with for some positive constant independent of . Under Assumption 3, we need the following two lemmas derived in (Shi et al., 2011; Shi, 2013) and (Wu and Zhou, 2005) to present the oracle inequalities.
Let be a set of functions on . For every , if almost everywhere and for some , and . Then for any ,
5.1 Oracle inequality for SVM with twice smooth classifying loss
We present the first oracle inequality, which describes the learning performance of SVM with a twice smooth classifying loss under Assumption 3.
Let be a twice smooth classifying loss. Under Assumption 3, if there exist constants and such that
then for arbitrary bounded , there holds
where and is a constant independent of , or whose value is specified in the proof and
To prove Theorem 4, we at first prove three propositions.
Let be defined by (16). Then for arbitrary , there holds
Proof. Direct computation yields
Since is a classifying loss, there holds . Then, it follows from (16) that
This finishes the proof of Proposition 1.
For any , if is a twice smooth classifying loss, then with confidence , there holds
Proof. Let . Since is continuous, we have . Hence, Moreover, , and the continuous differentiability of show that
Using Lemma 1 to the random variable , we obtain that
holds with confidence . This finishes the proof of Proposition 2.
Proof. Let . Set
For arbitrary there exists an such that . Therefore,
Applying Lemma 2 with , and to , we obtain
Since , it follows from the convexity and continuous differentiable property of that for arbitrary , there exist and such that
Thus, for any , an -covering of provides an -covering of . Therefore
Due to (16), we have