It is known in statistical machine learning and data mining that nonlinear algorithms can often achieve substantially better accuracies than linear methods, although typically nonlinear algorithms are considerably more expensive in terms of the computation and/or storage cost. The purpose of this paper is to compare the performance of 5 important nonlinear kernels and their corresponding linearization methods, to provide guidelines for practitioners and motivate new research directions.
We start the introduction with the basic linear kernel. Consider two data vectors. It is common to use the normalized linear kernel (i.e., the correlation):
This normalization step is in general a recommended practice. For example, when using LIBLINEAR or LIBSVM packages , it is often suggested to first normalize the input data vectors to unit
norm. The use of linear kernel is extremely popular in practice. In addition to packages such as LIBLINEAR which implement batch linear algorithms, methods based on stochastic gradient descent (SGD) become increasing important especially for very large-scale applications.
Next, we will briefly introduce five different types of nonlinear kernels and the corresponding randomization algorithms for linearizing these kernels. Without resorting to linearization, it is rather difficult to scale nonlinear kernels for large datasets . In a sense, it is not very practically meaningful to discuss nonlinear kernels without knowing how to compute them efficiently.
Note that in this paper, we restrict our attention to nonnegative data, which are common in practice. Several nonlinear kernels to be studied are only applicable to nonnegative data.
1.1 The acos Kernel
Consider two data vectors . The acos kernel is defined as a monotonic function of the correlation (1):
from the standard normal distribution and compute the inner products:
then the following probability relation holds:
If we generate independently such pairs of (
), we will be able to estimate the probability which approximates the acos kernel. Obviously, this is just a “pseudo linearization” and the accuracy of approximation improves with increasing sample size. In the transformed dataset, the number of nonzero entries in each data vector is exactly .
Specifically, we can encode (expand) (or ) as a 2-dim vector if and if . Then we concatenate such 2-dim vectors to form a binary vector of length . The inner product (divided by ) between the two new vectors approximates the probability .
1.2 The acos- Kernel
For the convenience of linearization via randomization, we consider the following acos- kernel:
As shown in , if we sample i.i.d.
from the standard cauchy distributionand again compute the inner product
then we obtain a good approximation (as extensively validated in ):
Again, we can encode/expand (or ) as a 2-dim vector if and if . In the transformed dataset, the number of nonzeros per data vector is also exactly .
1.3 Min-Max Kernel
The min-max (MM) kernel is also defined on nonnegative data:
where and is unbounded. See Appendix A for details. The basic theoretical result of CWS says
The recent work on “0-bit CWS”  showed that, by discarding , is a good approximation, which also leads to a convenient implementation. Basically, we can keep the lowest bits (e.g., or 8) of and view as a binary vector of length with exactly one 1. This way, the number of nonzeros per data vector in the transformed dataset is also exactly .
1.4 RBF Kernel and Folded RBF (fRBF) Kernel
The RBF (radial basis function) kernel is commonly used. For convenience (e.g., parameter tuning), we recommend this version:
where is the correlation defined in (1) and is a crucial tuning parameter.
This provides a mechanism for linearizing the RBF kernel.
It turns out that, one can simplify (11) by removing the need of . In this paper, we define the “folded RBF” (fRBF) kernel as follows:
which is monotonic in .
Suppose and and . Then the following identity holds:
Proof: See Appendix B.
1.5 Summary of Contributions
We propose the “folded RBF” (fRBF) kernel to simplify the linearization step of the traditional RBF kernel. Via our extensive kernel SVM experiments (i.e., Table 2), we show that the RBF kernel and the fRBF kernel perform similarly. Then through the experiments on linearizing RBF and fRBF kernels, both linearization schemes also perform similarly.
Our classification experiments on kernel SVM illustrate that even the best-tuned RBF/fRBF kernels in many datasets do not perform as well as those tuning-free kernels, i.e., the min-max kernel, the acos kernel, and the acos- kernel.
It is known that nonlinear kernel machines are in general expensive in computation and/or storage . For example, for a small dataset with merely data points, the kernel matrix already has entries. Thus, being able to linearize the kernels becomes crucial in practice. Our extensive experiments show that in general, the consistent weighted sampling (CWS) for linearizing the min-max kernel performs well, compared to randomization methods for linearizing the RBF/fRBF kernel, the acos kernel, or the acos- kernel. In particular, CWS usually requires only a relatively small number of samples to reach a good accuracy while other methods typically need a large number of samples.
We propose two new nonlinear kernels by combining the min-max kernel with the acos kernel or the acos- kernel. This idea can be generalized to create other types of nonlinear kernels.
The work in this paper suggests at least two interesting directions for future research: (i) To develop improved kernel functions. For example, the (tuning-free) min-max kernel in some datasets does not perform as well as the best-tuned RBF/fRBF kernels. Thus there is room for improvement. (ii) To develop better randomization algorithms for linearizing the RBF/fRBF kernels, the acos kernel, and the acos- kernel. Existing methods require too many samples, which means the transformed dataset will have many nonzeros per data vector (which will cause significant burden on computation/storage). Towards the end of the paper, we report our proposal of combining the min-max kernel with the acos kernel or the acos- kernel. The initial results appear promising.
2 An Experimental Study on Kernel SVMs
|Dataset||# train||# test||# dim||linear (%)|
Table 1 lists the 35 datasets for our experimental study in this paper. These are the same datasets used in a recent paper  on the min-max kernel and consistent weighted sampling (0-bit CWS). The last column of Table 1 also presents the best classification results using linear SVM.
Table 2 summarizes the classification results using 5 different kernel SVMs: the min-max kernel, the RBF kernel, the fRBF kernel, the acos kernel, and the acos- kernel. More detailed results (for all regularization values) are available in Figures 1 to 3. To ensure repeatability, for all the kernels, we use the LIBSVM pre-computed kernel functionality. This also means we can not (easily) test nonlinear kernels on larger datasets, for example, “WebspamN1” in the last row of Table 1.
For both RBF and fRBF kernels, we need to choose , the important tuning parameter. For all the datasets, we exhaustively experimented with 58 different values of 0.001, 0.01, 0.1:0.1:2, 2.5, 3:1:20 25:5:50, 60:10:100, 120, 150, 200, 300, 500, 1000. Here, we adopt the MATLAB notation that (e.g.,) 3:1:20 means all the numbers from 3 to 20 spaced at 1. Basically, Table 2 reports the best RBF/fRBF results among all and values in our experiments.
|Covertype10k||80.4||80.1 (120)||80.1 (100)||81.9||81.6|
|Covertype20k||83.3||83.8 (150)||83.8 (150)||85.3||85.0|
|IJCNN5k||94.4||98.0 (45)||98.0 (40)||96.9||96.6|
|IJCNN10k||95.7||98.3 (60)||98.2 (50)||97.5||97.4|
|Isolet||96.4||96.8 (6)||96.9 (11)||96.5||96.1|
|Letter||96.2||97.6 (100)||97.6 (100)||97.0||97.0|
|Letter4k||91.4||94.0 (40)||94.1 (50)||93.3||93.3|
|M-Basic||96.2||97.2 (5)||97.2 (5)||95.7||95.8|
|M-Image||80.8||77.8 (16)||77.8 (16)||76.2||75.2|
|MNIST10k||95.7||96.8 (5)||96.9 (5)||95.2||95.2|
|M-Noise1||71.4||66.8 (10)||66.8 (10)||65.0||64.0|
|M-Noise2||72.4||69.2 (11)||69.2 (11)||66.9||65.7|
|M-Noise3||73.6||71.7 (11)||71.7 (11)||69.0||68.0|
|M-Noise4||76.1||75.3 (14)||75.3 (14)||73.1||71.1|
|M-Noise5||79.0||78.7 (12)||78.6 (11)||76.6||74.9|
|M-Noise6||84.2||85.3 (15)||85.3 (15)||83.9||82.8|
|M-Rand||84.2||85.4 (12)||85.4 (12)||83.5||82.3|
|M-Rotate||84.8||89.7 (5)||89.7 (5)||84.5||84.6|
|M-RotImg||41.0||45.8 (18)||45.8 (18)||41.5||39.3|
|Optdigits||97.7||98.7 (8)||98.7 (8)||97.7||97.5|
|Pendigits||97.9||98.7 (13)||98.7 (11)||98.3||98.1|
|Phoneme||92.5||92.4 (10)||92.5 (9)||92.2||90.2|
|Protein||72.4||70.3 (4)||70.2 (4)||69.2||70.5|
|RCV1||96.9||96.7 (1.7)||96.7 (0.3)||96.5||96.7|
|Satimage||90.5||89.8 (150)||89.8 (150)||89.5||89.4|
|Segment||98.1||97.5 (15)||97.5 (15)||97.6||97.2|
|SensIT20k||86.9||85.7 (4)||85.7 (4)||85.7||87.5|
|Shuttle1k||99.7||99.7 (10)||99.7 (15)||99.7||99.7|
|Spam||95.0||94.6 (1.2)||94.6 (1.7)||94.2||95.2|
|Splice||95.2||90.0 (15)||89.8 (16)||89.2||91.7|
|USPS||95.3||96.2 (11)||96.2 (11)||95.3||95.5|
|Vowel||59.1||65.6 (20)||65.6 (20)||63.0||61.3|
|WebspamN1-20k||97.9||98.0 (35)||98.0 (35)||98.1||98.5|
|YoutubeVision||72.2||70.2 (3)||70.1 (4)||69.6||74.4|
Table 2 shows that RBF kernel and fRBF kernel perform very similarly. Interestingly, even with the best tuning parameters, RBF/fRBF kernels do not always achieve the highest classification accuracies. In fact, for about of the datasets, the min-max kernel (which is tuning-free) achieves the highest accuracies. It is also interesting that the acos kernel and the acos- kernel perform reasonably well compared to the RBF/fRBF kernels.
Overall, it appears that the RBF/fRBF kernels tend to perform well in very low dimensional datasets. One interesting future study is to develop new kernel functions based on the min-max kernel, the acos kernel, or the acos- kernel, to improve the accuracies. The new kernels could be the original kernels equipped with a tuning parameter via a nonlinear transformation. One challenge is that, for any new (and tunable) kernel, we must also be able to find a randomization algorithm to linearize the kernel; otherwise, it would not be too meaningful for large-scale applications.
3 Linearization of Nonlinear Kernels
It is known that a straightforward implementation of nonlinear kernels can be difficult for large datasets . As mentioned earlier, for a small dataset with merely data points, the kernel matrix has entries. In practice, being able to linearize nonlinear kernels becomes very beneficial, as that would allow us to easily apply efficient linear algorithms in particular online learning . Randomization is a popular tool for kernel linearization.
Since LIBSVM did not implement most of the nonlinear kernels in our study, we simply used the LIBSVM pre-computed kernel functionality in our experimental study as reported in Table 2. While this strategy ensures repeatability, it requires very large memory.
In the introduction, we have explained how to linearize these 5 types of nonlinear kernels. From practitioner’s perspective, while results in Table 2 are informative, they are not sufficient for guiding the choice of kernels. For example, as we will show, for some datasets, even though the RBF/fRBF kernels perform better than the min-max kernel in the kernel SVM experiments, their linearization algorithms require many more samples (i.e., large ) to reach the same accuracy as the linearization method (i.e., 0-bit CWS) for the min-max kernel.
3.1 RBF Kernel versus fRBF Kernel
We have explained how to linearize both the RBF kernel and the fRBF kernel in Introduction. For two normalized vectors , we generate i.i.d. samples and independent . Let and . Then we have
In order to approximate the expectations with sufficient accuracies, we need to generate the samples many (say ) times. Typically has to be large. In our experiments, even though we use as large as 4096, it looks we will have to further increase in order to reach the accuracy of the original RBF/fRBF kernels (as in Table 2).
Figure 4 reports the linear SVM experiments on the linearized data for 10 datasets, for as large as 4096. We can see that, for most datasets, the linearized RBF and linearized fRBF kernels perform almost identically. For a few datasets, there are visible discrepancies but the differences are small. We repeat the experiments 10 times and the reported results are the averages. Note that we always use the best values as provided in Table 2.
3.2 Min-max Kernel versus RBF/fRBF Kernels
Table 2 has shown that for quite a few datasets, the RBF/fRBF kernels outperform the min-max kernel. Now we compare their corresponding linearization algorithms. We adopt the 0-bit CWS  strategy and use at most 8 bits for storing each sample. See the Introduction and Appendix A for more details on consistent weighted sampling (CWS).
Figure 5 compares the linearization results of the min-max kernel with the results of the RBF kernel. We can see that the linearization algorithm for RBF performs very poorly when the sample size is small (e.g., ). Even with , the accuracies still do not reach the accuracies using the original RBF kernel as reported in Table 2.
There is an interesting example. For the “M-Rotate” dataset, the original RBF kernel notably outperforms the original min-max kernel ( versus ). However, as shown in Figure 5, even with 4096 samples, the accuracy of the linearized RBF kernel is still substantially lower than the accuracy of the linearized min-max kernel.
These observations motivate a useful future research: Can we develop an improved linearization algorithm for RBF/fRBF kernels which would require much fewer samples to reach good accuracies?
3.3 Min-max Kernel versus acos and acos- Kernels
As introduced at the beginning of the paper, sign Gaussian random projections and sign Cauchy random projections are the linearization methods for the acos kernel and the acos- kernel, respectively. Figures 6 and 7 compare them with 0-bit CWS, where we use “” for sign Gaussian projections and “” for sign Cauchy projections.
Again, like in Figure 5, we can see that the linearization method for the min-max kernel requires substantially few samples than the linearization methods for the acos and acos- kernels. Since both kernels show reasonably good performance (without linearization), This should also motivate us to pursue improved linearization algorithms for the acos and acos- kernels, as future research.
3.4 Comparisons on a Larger Dataset
Figure 8 provides the comparison study on the “WebspamN1” dataset, which has 175,000 examples for training and 175,000 examples for testing. It is too large for using the LIBSVM pre-computed kernel functionality in common workstations. On the other hand, we can easily linearize the nonlinear kernels and run LIBLINEAR on the transformed dataset.
The left panel of Figure 8 compares the results of linearization method (i.e., 0-bit CWS) for the min-max kernel with the results of the linearization method for the RBF kernel. The right panel compares 0-bit CWS with sign Gaussian random projections (i.e., ). We do not present the results for since they are quite similar. The plots again confirm that 0-bit CWS significantly outperforms the linearization methods for both the RBF kernel and the acos kernel.
4 Kernel Combinations
It is an interesting idea to combine kernels for better (or more robust) performance. One simple strategy is to use multiplication of kernels. For example, the following two new kernels
combine the min-max kernel with the acos kernel or the acos- kernel. They are still positive definite because they are the multiplications of positive definite kernels.
Table 3 presents the kernel SVM experiments for these two new kernels (i.e., the last two columns). We can see that for majority of the datasets, these two kernels outperform the min-max kernel. For a few datasets, the min-max kernel still performs the best (for example, “M-Noise1”); and on these datasets, the acos kernel and the acos- kernel usually do not perform as well. Overall, these two new kernels appear to be fairly robust combinations. Of course, the story will not be complete until we have also studied their corresponding linearization methods.
A recent study  explored the idea of combing the “resemblance” kernel with the linear kernel, designed only for sparse non-binary data. Since most of the datasets we experiment with are not sparse, we can not directly use the special kernel developed in .
Now we study the linearization methods for these two new kernels, which turn out to be easy. Take the MM-acos kernel as an example. We can separately and independently generate samples for the min-max kernel and the acos kernel. The sample for the min-max kernel can be viewed as a binary vector with one 1. For example, if the sample for the min-max kernel is and the sample for the acos kernel is -1. Then we can encode the combined sample as . If the sample for the acos kernel is 1, then the combined vector becomes . Basically, if the -th location in the vector corresponding to the original min-max sample is 1, then the combined vector will double the length and all the entries will be zero except the (-1)-th or ()-th location, depending on the sample value of acos kernel.
Clearly, the idea also applies for combining min-max kernel with RBF kernel. We just need to replace the “1” in the vector for the min-max kernel sample with the sample of the RBF kernel.
Figure 9 and Figure 10 report the linear SVM results using linearized data for the MM-acos kernel (right panels) and the MM-acos- kernel (left panels), to compare with the results using linearized data for the min-max kernel (solid curves). We can see that the linearization methods for the MM-acos kernel and the MM-acos- kernel outperform the linearization method for the min-max kernel when is not large. These preliminary results are encouraging.
Nonlinear kernels can be potentially very useful if there are efficient (in both storage and memory) algorithms for computing them. It has been known that the RBF kernel, the acos kernel, and the acos- kernel can be linearized via randomization algorithms. There are two major aspects when we compare nonlinear kernels: (i) the accuracy of the original kernel; (ii) how many samples are needed in order to reach a good accuracy. In this paper, we try to address these two issues by providing an extensive empirical study on a wide variety of publicly available datasets.
To simplify the linearization procedure for the RBF kernel, we propose the folded RBF (fRBF) kernel and demonstrates that its performance (either with the original kernel or with linearization) is very similar to that of the RBF kernel. On the other hand, our extensive nonlinear kernel SVM experiments demonstrate that the RBF/fRBF kernels, even with the best-tuned parameters, do not always achieve the best accuracies. The min-max kernel (which is tuning-free) in general performs well (except for some very low dimensional datasets). The acos kernel and the acos- kernel also perform reasonably well.
Linearization is a crucial step in order to use nonlinear kernels for large-scale applications. Our experimental study illustrates that the linearization method for the min-max kernel, called “0-bit CWS”, performs well in that it does not require a large number of samples to reach a good accuracy. In comparison, the linearization methods for the RBF/fRBF kernels and the acos/acos- kernels typically require many more samples (e.g., ).
Our study motivates two interesting research problems for future work: (i) how to design better (and still linearizable) kernels to improve the tuning-free kernels; (ii) how to improve the linearization algorithms for the RBF/fRBF kernels as well as the acos/acos- kernels, in order to reduce the required sample sizes. The interesting and simple idea of combing two nonlinear kernels by multiplication appears to be effective but we still hope to find an even better strategy in the future.
Another challenging task is to develop (linearizable) kernel algorithms to compete with (ensembles of) trees in terms of accuracy. It is known that tree algorithms are usually slow. Even though the parallelization of trees is easy, it will still consume excessive energy (e.g., electric power). One can see from [12, 13] that trees are in general perform really well in terms of accuracy and can be remarkably more accurate than other methods in some datasets (such as “M-Noise1” and “M-Image”). On top of the fundamental works [7, 6], the recent papers [12, 13]
improved tree algorithms via two ideas: (i) an explicit tree-split formula using 2nd-order derivatives; (ii) a re-formulation of the classical logistic loss function which leads to a different set of first and second derivatives from textbooks. Ideally, it would be great to develop statistical machine learning algorithms which are as accurate as (ensembles of) trees and are as fast as linearizable kernels.
Appendix A Consistent Weighted Sampling
Appendix B Proof of Lemma 1
Let . Using the bivariate normal density function, we obtain
-  L. Bottou. http://leon.bottou.org/projects/sgd.
-  L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors. Large-Scale Kernel Machines. The MIT Press, Cambridge, MA, 2007.
O. Chapelle, P. Haffner, and V. N. Vapnik.
Support vector machines for histogram-based image classification.
IEEE Transactions on Neural Networks, 10(5):1055–1064, 1999.
-  M. S. Charikar. Similarity estimation techniques from rounding algorithms. In STOC, pages 380–388, Montreal, Quebec, Canada, 2002.
-  R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. Liblinear: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874, 2008.
J. H. Friedman.
Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 29(5):1189–1232, 2001.
J. H. Friedman, T. J. Hastie, and R. Tibshirani.
Additive logistic regression: a statistical view of boosting.The Annals of Statistics, 28(2):337–407, 2000.
-  M. X. Goemans and D. P. Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of ACM, 42(6):1115–1145, 1995.
-  T. J. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning:Data Mining, Inference, and Prediction. Springer, New York, NY, 2001.
-  S. Ioffe. Improved consistent sampling, weighted minhash and L1 sketching. In ICDM, pages 246–255, Sydney, AU, 2010.
-  H. Larochelle, D. Erhan, A. C. Courville, J. Bergstra, and Y. Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In ICML, pages 473–480, Corvalis, Oregon, 2007.
-  P. Li. Abc-boost: Adaptive base class boost for multi-class classification. In ICML, pages 625–632, Montreal, Canada, 2009.
-  P. Li. Robust logitboost and adaptive base class (abc) logitboost. In UAI, 2010.
-  P. Li. CoRE kernels. In UAI, Quebec City, CA, 2014.
-  P. Li. 0-bit consistent weighted sampling. In KDD, Sydney, Australia, 2015.
-  P. Li, G. Samorodnitsky, and J. Hopcroft. Sign cauchy projections and chi-square kernel. In NIPS, Lake Tahoe, NV, 2013.
-  M. Manasse, F. McSherry, and K. Talwar. Consistent weighted sampling. Technical Report MSR-TR-2010-73, Microsoft Research, 2010.
-  A. Rahimi and B. Recht. andom features for large-scale kernel machines. In NIPS, 2007.
-  W. Rudin. Fourier Analysis on Groups. John Wiley & Sons, New York, NY, 1990.
-  B. Schiele and J. L. Crowley. Object recognition using multidimensional receptive field histograms. In ECCV, pages 610–619, Helsinki, Finland, 1996.