share
1 Introduction
The support vector machine (SVM) Vapnik98 ; Vapnik99
is a high performance learning algorithm constructing a hyperplane to separate two-class data by maximizing the margin between them. There are two approaches for extending SVMs to multi-class problems, i.e., solving the problem by formulating all classes of data under a single optimization, and combining several two-class subproblems. However, the difficulty and complexity to solve the problem with the first method are due to the increase of the number of classes and the size of training data, so the second method is more suitable for practical use. In this paper, we focus on the second approach.
For constructing a multi-class classifier from binary ones, the method called one-against-one trains each binary classifier on only two out of classes, and builds possible classifiers. Several strategies have been proposed for combining the trained classifiers to make the final classification for an unseen data. Friedman Friedman96 suggested the combination strategy called Max Wins. In the classification process of Max Wins, every binary classifier provides one vote for its preferred class and the class with the largest vote will be set to be the final output. Chang and Lee Chang11 investigated an adaptive framework to manage a nuisance vote which is a vote for an unrelated class by allowing a classifier to make a non-vote for data of unrelated class. Instead of a binary classifier, they employed a ternary classifier that consists of two particular classes and the rest of the classes fused as the third class for solving this problem.
Vapnik Vapnik98 proposed the one-against-the-rest approach working by constructing a set of binary classifiers in which each classifier is learned from all examples in the class, and the remaining classes labeled with the positive and negative classes, respectively. The class corresponding to the classifier with the highest output value is used to make the final output. Moreover, Manikandan and Venkataramani Manikandan10 adapted the traditional one-against-the-rest to work as a sequential classifier. All classifiers will be ordered corresponding to their misclassification. This method needs a lower number of classifiers on avearge compared with the traditional one-against-the-rest, but both algorithms have the same problem in the training phase because of the difficulty for calculating the absolutely separating hyperplane between a class and all of the other classes.
Dietterich and Bakiri Dietterich95 introduced the Error Correcting Output Code (ECOC) based on the fundamental of information theory. For a given code matrix with rows and columns, each element contains either 1’, or -1’. Each column denotes the bit string showing the combination of positive and negative classes for constructing a binary classifier, and each row of the code matrix indicates the unique bit string for representing a specific class (each bit string is called a codeword). Allwein et al. Allwein2000 extended the coding method by adding the third symbol 0’ as “ don’t care bit” to allow the binary model learned without considering some particular classes. Unlike the previous method, the number of classes for training a binary classifier can be varied from 2 to classes. Based on these two systems for an classes problem, the maximum numbers of different binary classifiers are Dietterich95 , and Bagheri2013 , respectively. Design of code matrices with different subsets of binary classifiers gives different abilities for separating classes, and the problem of selecting a suitable subset of binary classifiers is complicated with a large size of
. To obtain the suitable code matrix, some techniques using the Genetic Algorithm have been proposed
Kuncheva2005 ; Lorena2009 . In the classification phase, a test example is classified by all classifiers corresponding to the column of the code matrix, and then the class with the closet codeword is assigned to the final output class.Platt et al. Platt99 proposed the Decision Directed Acyclic Graph (DDAG) in order to reduce evaluation time Hsu02 . In each round, a binary model will be randomly selected from all classifiers. The binary classification result is employed to eliminate the candidate output classes, and to ignore all binary classifiers related to the defeated class. It guarantees that the number of classifications (applied classifiers) of the DDAG is always . This recursive task will be applied until there is only one last candidate class. However, the misclassification of the DDAG can be occurred at the time when selected binary classifiers related to the target class (hence forth BCRT) give the wrong answer. The more times the number of BCRTs are applied, the more chance the misclassification is produced by the DDAG. In order to reduce this risk, Kijsirikul and Ussivakul Kijsirihul02 proposed the Adaptive Directed Acyclic Graphs (ADAG) that has a reversed triangular structure of the DDAG. It requires only times or less that the target class is tested against the other classes, while the DDAG possibly requires at most times.
In addition, there have been many attempts that apply some information such as the margin size Platt99 , the number of support vectors Abe03 , and the separability measures between classes Lorena11 ; Li11 , to improve the performance of the multi-class classification. The margin size and the number fo support vectors were applied for selecting the suitable two-class classifiers in the DDAG Platt99 ; Abe03 . The separability measure was employed for automatically constructing a binary tree of multi-class classification based on the concept of the minimum spanning tree Lorena11 . Li, et al. Li11 used similar information to vote the preferred class for data in unclassifiable region for both the one-against-one and the one-against-the-rest techniques.
In this research, we investigate the framework for enhancing three well-known methods, which are the DDAG, the ADAG, and Max Wins. Max Wins is currently recognized as the-state-of-the-art combining algorithm and it is also the most powerful technique among all of our focused works with a need of number of classifications for an -class problem, while the other two approaches reduce the number of classifications to . We study the characteristics of these methods that lead to wrong classification results. The first two techniques have the same hierarchical structure and have the same weak point that they “trust on individual opinion” for making decision to discard the candidate classes. Intuitively, if only one of BCRTs makes a mistake, the whole system will give the wrong output. The last technique as the high performance one, Max Wins is based on the concept of “trust on most popular opinion” for making decision to select the output class. If all of BCRTs give the correct answer, Max Wins will always provide the correct output class. However, if there exists only one of BCRTs give the wrong answer, it may lead to misclassification due to equal voting, or other non-target classes reaching the largest vote as shown later in the paper. Examples which are incorrectly classified in this scenario can be recovered by our proposed strategies.
In this paper, we demonstrate that the above traditional methods can be improved based on the same idea that if we access further important information of generalization performance
of all binary classifiers and properly estimate it, it can be employed for enhancing the performance of the methods. Based on this idea, we propose four novel approaches including (1) the Reordering Adaptive Directed Acyclic Graph (RADAG), (2) Strong Elimination of the classifiers (SE), (3) Weak Elimination of the classifiers (WE), and (4) Voting based Candidate Filtering (VCF). The first approach, the next two approaches, and the last approach are improved from the ADAG, the DDAG, and Max Wins, respectively. We also empirically evaluate our methods by comparing them with the traditional methods on the sixteen datasets from the UCI Machine Learning Repository
Blake98 .This paper is organized as follows. Section 2 reviews the traditional multi-class classification frameworks. Section 3 describes how to properly estimate the generalization performance of binary classifiers. Section 4 presents our proposed methodologies. Section 5 performs experiments and explains the results and discussions. Section 6 concludes the research.
2 Multi-class Support Vector Machines
2.1 Max Wins
For an -class problem, all possible pairs of two-class data are learned for constructing classifiers. All binary classifiers are applied for voting the preferred class. A class with maximum vote will be assigned as the final output class. This method is called Max Wins Friedman96 . However, in case that there exists more than one class giving the same maximum vote, the final output class can be obtained by random selection from candidate classes with the equal maximum-vote. An example of the classification using this technique for a four-class problem is shown in Fig. 1. Each class will be voted (solid-line) or ignored (dash-line) by all related binary models. For example, class has three related classifiers, i.e., 1 vs 2, 1 vs 3, and 1 vs 4. The voting result of class , class , class and class are three, zero, one, and, two, respectively. In this case, class has the largest score, and therefore it is assigned as the final output class.
2.2 Decision Directed Acyclic Graphs
Platt et al. Platt99 introduced a learning algorithm using the Directed Acyclic Graph (DAG) to represent the classification task, called the Decision Directed Acyclic Graph (DDAG). This architecture represents a set of nodes connected by edges with no cycles. Each edge has an orientation and each node has either 0 or 2 edges. Among these nodes, there exists a root node which is the unique node with no edge pointing into it. In a DDAG, the nodes are arranged with a triangular shape in which each node is labeled with an element of a boolean function. There exists a single root node at the top, two nodes in the second layer, and so on until the final layer of leaves for an -class problem.
To make a classification, an example with an unknown class label is evaluated by the nodes as binary decision functions. The binary output result in each layer is applied to eliminate the candidate output classes and the binary classifiers related to the defeated class are removed. At the first layer (see Fig. 2), the root node can be randomly selected from all possible classifiers and there are candidate output classes. After the root node is tested, its binary result is employed to eliminate the candidate output classes and the binary classifiers corresponding to the defeated class are discarded. In the next layer, the remaining binary classifiers are randomly selected to continue the same process in which some classes are eliminated from the remaining candidate classes. The process is repeated until there is only one class remained which is then assigned as the final output class. This algorithm requires only decision nodes in order to obtain the final answer.
One disadvantage of the DDAG is that its classification result is affected by the sequence of binary classifiers randomly selected in the evaluation process. Platt et al. also proposed the other method that prefers the binary decision function with the higher generalization performance measured by its margin sizes, called the large margin DAGs Platt99 . The margin size () is a parameter for bounding the generalization ability of the binary SVM as shown in terms of the VC-dimension in Eq. (2). It illustrates that the generalization performance of the binary model is proportional to the size of the margin. A binary classifier with the larger margin size will be firstly applied in each round of the evaluation step. Moreover, Takahashi and Abe Abe03 proposed a similar framework that employed the number of support vectors as a performance measure. In this method, the generalization error () for classes and was bounded by Eq. (1) Vapnik95 :
| (1) |
where is the number of support vectors for classes and and is the number of training data for classes and .
2.3 Adaptive Directed Acyclic Graphs
In the DDAG, binary classification result of a previously employed binary classifier is used to eliminate a candidate output classe, and there are only current remaining candidate classes that can be possibly assigned as the final output class. Therefore, the misclassification of a selected BCRT is the crucial point.
The ADAG was originally designed to reduce this risk of the DDAG by using reversed triangular structure Kijsirihul02 . In an -class problem, there are nodes at the top, nodes in the second layer and so on until the lowest layer of the final node, as illustrated in Fig. 3. Like the DDAG, binary output results of the ADAG in each layer are applied to discard candidate output classes and the binary classifiers related to the defeated classes are also ignored. Therefore, the ADAG also evaluates only nodes to obtain the final answer.
According to the critical issue of misclassification mentioned above, even only one selected classifier related to the target class provides a wrong answer, the misclassification on the final output class cannot be avoided. Hence, the number of times the target class is tested against other classes indicates the risk of misclassification. The DDAG requires at most times that the target class is tested against other classes, while the ADAG requires only times or less. This shows that the opportunities of the target class tested against other classes on the ADAG is much lower than the DDAG.
3 An Estimation of the Generalization Performan-ces of Binary Support Vector Machines
The generalization performance of a learning model is the actual performance evaluated on unseen data. For support vector machines, a model is trained by using the concept of the Structure Risk Minimization principle Vapnik74 in which the generalization performance of the model is estimated based on both terms of the complexity of model (the VC dimension of approximating functions) and the quality of fitting training data (empirical error). Consider the problem of binary classification where dataset of samples in real -dimensional space is randomly independent identically distributed observations drawn according to . The expected risk (
) with probability at least
can be bounded by the following equation Bartlett99 ; Burges98 :| (2) |
where there is a corresponding constant
for all probability distributions,
is the number of labeled examples in with margin less than , indicates the radius of the smallest sphere that contains all the data points, and is the distance between the hyperplane and the closest points of the training set (margin size). The first and second terms of inequality in Eq. (2) denote the bound of the empirical error, and the VC dimension, respectively.In our frameworks, the generalization ability will be applied to improve the multi-class classification. Although there have been many attempts to use some performance measures such as the margin size Platt99 , the number of support vectors Abe03 , they may not accurately reflect the actual performance of each binary SVM. Consider a two-class problem where hyperplanes and are learning models created to separate the positive and the negative examples. Suppose that they provide different margin sizes of and , and the different numbers, and , of labeled examples in with the margin less than their margin sizes, respectively. In case that the parameters and are fixed, there are only two parameters including and that affect the performance of the learning model (as the parameters and , as well as and are the same for the same pair of a two-class problem). Now consider two-learning models learned from different pairs of a two-class problem. In case that the parameters , and are fixed, according to inequality in Eq. (2), obviously, if we use only , , or combination of them, they are not sufficient to represent the whole term of their generalization abilities. This shows that a binary model with the larger margin size may not provide more accurate result of classification. The use of only the number of support vectors is also shown in Burges98 that it is not predictive for generalization ability.
|
|
|
|
As described above, the generalization ability can be employed to enhance the performance of multi-class classification, by carefully design algorithms which utilize this information as a selection measure for good classifiers. We believe that the generalization performance of binary SVMs can be directly estimated by -fold cross-validation Mitchell97 (see Algorithm 1), and it can be used to fairly compare the performances of binary SVMs on different two-class problems. Below we give an example which demonstrates that -fold cross-validation is more suitable for estimating the generalizaiton performance of the classifiers than the other measures used by the previous methods, i.e. the number of support vectors, the margin size.
Fig. 4 shows the generalization performance measured by the previous methods Platt99 ; Abe03 , and -fold cross-validation, which we propose to use as the performance measure, for the Letter dataset with 26 classes, by applying the polynomial kernel of . Fig. 4 (a) illustrates that the trend of estimated generalization error by -fold cross-validation is very closed to the actual risk, while the other two techniques give high variation. To further investigation in more details, we select about of all classifiers to show in Fig. 4 (b-d); these figures illustrate the comparisons between the actual risk and the estimated generalization errors with different measures, i.e., CV Bound, SV Bound, and Normalized Margin Bound, respectively. In each figure, classifiers are sorted in ascending order by the estimated generalization errors. It is expected that if a specific measure is a good estimator for generalization error, its value should be in the same trend as the actual risk (its value shoud increase with the increase of the actual risk). A good trend is found in Fig. 4 (b), while the other two methods give no clear trend and contain confusing patterns. In order to evaluate the efficiency of these estimating methods, we apply the correlation analysis between two variables Johnson2001 , i.e., the actual risks and these three estimated generalization errors. These evaluations are based on 325 classifiers as in Fig. 4 (a), and the statistical r-values of them are 0.805, 0.372, and -0.230 as shown in Fig. 4 (b-d), respectively. The r-values also confirm that CV Bound and actual risk have high correlation, while the other two methods give very low correlation. They show that -fold cross-validation is more suitable to be the measure for the performance of binary classifiers. According to the above reason, we apply this measure in our research.
4 The proposed methods
The combination of binary SVMs with high generalization performance directly affects the accuracy of the multi-class classification. In this section, we introduce four enhanced approaches based on the previous techniques i.e., the ADAG, the DDAG, and Max Wins by applying the generalization abilities in order to select suitable binary classifiers. An improvement of the ADAG is called the Reordering Adaptive Directed Acyclic Graph (RADAG). There are two improved versions for the DDAG i.e., Strong Elimination of the classifiers (SE) and Weak Elimination of the classifiers (WE). The last technique is Voting-based Candidate Filtering (VCF) enhanced from Max Wins. To increase the classification accuracy, the generalization estimated by -fold cross-validation is utilized as the goodness measure of classifiers in our frameworks.
4.1 Reordering Adaptive Directed Acyclic Graph
The ADAG is designed to reduce the number of times the binary classifiers related to the target class are applied, from at most times required by the DDAG, to times or less. However, binary classifiers in the first level of the ADAG are still randomly selected, and its misclassification can be produced at the time even when only one BCRT gives a wrong answer. In this section, we introduce a more effective method which uses the minimum weight perfect matching to select the optimal pairs of classes in each level with minimum generalization error. We called the method the Reordering Adaptive Directed Acyclic Graph (RADAG).
The structure of the RADAG is similar to the ADAG, but they are different in the initialization of the binary classifiers in the top level and the order of classes in lower levels (see Fig. 4.1). The reordering algorithm with minimum weight perfect matching is described in Algorithm 2. The algorithm selects the optimal order of classes in each level. It is different from the ADAG in that the initial order of classes in the ADAG is obtained randomly, and the matching of classes in successive levels depends on the classification results of nodes from the previous level. For the RADAG, the reordering process will be applied to the remaining candidate classes in all levels for determining the optimal sequence of them.
To select the optimal set of classifiers, the generalization measure in Section 3 is used as a criterion. This scheme provides less chance to predict the wrong class from all possible orders. Among classifiers, classifiers which have the smallest sum of generalization errors will be used in the classification.
Let be a graph with node set and edge set . Each node in denotes one class and each edge indicates one binary classifier of which generalization error is estimated from Section 3 (see Fig. 6(a)). The output of the reordering algorithm for graph is a subset of edges with the minimum sum of generalization errors of all edges and each node in is met by exactly one edge in the subset (see Fig. 6(b)).
Given a real weight being generalization error for each edge of , the problem of reordering algorithm can be solved by the minimum weight perfect matching Cook97 that finds a perfect matching of minimum weight .
For , let .
is the set of edges with both endpoints in . The set of edges incident to
node in the node-edge incidence matrix is denoted by .
The convex hull
of perfect matchings on a graph with even is given by
a)
b) for
c)
for all odd sets
with or by (a),(b) andd) for all odd sets with
where , and () means that is (is not) in the matching.
Hence, the minimum weight of a perfect matching is at least as large as the value of
| (3) |
where satisfies “(a), (b), and (c)” or “(a), (b) and, (d)”. Therefore, the reordering problem can be solved by the integer program in Eq. (3).
4.2 Strong & Weak Elimination of Classifiers for Enhancing Decision Directed Acyclic Graph
According to the characteristic of the DDAG, binary classification results of the previously employed binary classifiers are used to eliminate the candidate output classes, and thus the final output class will be assigned with one of the remaining candidate classes. By using the random technique for selecting a binary classifier, the DDAG produces mis-classification at the time when a BCRT with very low performance is selected and provides the wrong answer, as the target class will be discarded from the remaining candidate classes, and it is not possible to reach the correct output class. In this section, we propose the framework to enhance the performance of the DDAG to select the binary classifier with high performance based on the generalization abilities of binary classifiers as described in Section 3.
We propose two methods that are Strong Elimination of the classifiers (SE) and Weak Elimination of the classifiers (WE). Both algorithms are described in Algorithm 3 and Algorithm 4. We also show a classification process of SE and WE for an -class problem in Fig. 7 and Fig. 8, respectively.
For both of the DDAG and SE, in each round, a defeated class will be removed from candidate output classes, and all binary classifiers related to the defeated class are ignored. Due to this reason, they guarantee number of classifications for an -class problem. However, these ignored classifiers may have high generalization abilities and thus are helpful to eliminate the other remaining candidate classes. Therefore, we then propose WE to make use of binary classifiers with high generalization abilities.
According to the classifier elimination of WE, the number of classifications is bounded with the best case of , and the worst case of . However, WE provides the opportunities to employ better classifiers as shown in the Fig. 8. At round , suppose that classifier vs has lower generalization error than classifier vs , and both of them are active classifiers. In this case, it is possible that classifier vs can remove the class from the list of two remaining candidate classes, and can avoid using classifier vs with lower reliability that is unavoidable for SE as shown in Fig. 7.
4.3 Voting Based Candidate Filtering
Max Wins is one of high performance techniques that work based on the concept of “trust on the most popular opinion” for making decision to select the output class. If all of BCRTs give the correct answer, Max Wins will always provide the correct output class. It does not depend on the answers of the other binary classifiers. However, if only one of BCRTs gives a wrong answer, it may lead to misclassification due to equal voting, or another non-target class reaching the largest vote. Fig. 9 shows an example of such cases, taken from our experiment on the Letter dataset (see Section for more details); Fig. 9(a) and (b) show the cases of equal voting and another non-target class having the largest vote, respectively.
We propose a novel multi-class classification approach that alleviates the above problem of Max Wins, and uses the same concept “trust on the most popular opinion” for filtering out the low competitive classes. On the other hand, high competitive classes will be voted to be candidate output classes, though there exist some BCRTs providing the wrong answer. If there is more than one remaining class, the output class will be selected via the mechanism of WE. Our proposed technique aims to combine the strong point of both Max Wins and WE, and is called Voting based Candidate Filtering (VCF). The details of our algorithm are shown in Algorithm 5.
Let , and indicate the maximum of scores for all classes, and the score of class for a test data, respectively. Also let denotes the percentage of the difference between and . An example of the calculation of is shown in Fig. 9 (a), where E’, the score of class E’ = 23 points, and the score of class C’ = 24 points (as the top score). Then value can be calculated by . We also define to be the threshold of for considering class as a candidate for the target class; class will be accepted into the set of high competitive candidate classes if and only if its is less than or equal to . We want to keep the size of the filtered candidate classes as small as possible while still containing the target class.
|
|
A case study of high risk of misclassification in the Letter dataset including 4,010 examples where Max Wins provides 3,549 examples with the correct result, and 461 examples with high risk of misclassification. By a high-risk example, we mean (1) the example with an equal vote (the score of the target class is equal to those of other non-target classes) and (2) the example with a vote less than the maximum vote that is then mis-classified by Max Wins. These high risk examples will be hopefully recovered with the correct class label by our proposed algorithm. In our experiment, the high-risk examples includes 24 examples (around 5) with an equal vote, and 437 examples (around 95) with a vote less than the maximum as shown in Fig. 10, where represents the percentage of the difference between and the score of the target class. For each example, we calculate the rank of the voting score of the target class compared to the other non-target classes, and consider only the first eight ranks. There are 24 examples (around ) in the first rank, while in the second to the eighth ranks, the numbers of examples are 171, 77, 31, 30, 15, 14, and 10 (around , and ), respectively. The examples with the different ranks have different ranges of values, such as, in the second rank, the values are varied from to , in the third rank, the values are varied from to , in the fourth rank, the values are varied from to , and so on.
According to this case study, there can be at most of examples that will be correctly classified with the correct class label by random selection of Max Wins, while the other of examples will be absolutely misclassified. We want to recover an example that is not correctly classified by Max Wins, as its actual target class is not in the first rank or its target class has equal vote with some other output classes. If is set as in the VCF algorithm, it will guarantee that all high-risk misclassified examples with values no greater than can be filtered into the set of the candidate output classes; in this case only the examples in the first rank ( of examples) will be selected. When we apply a bigger threshold, e.g. , it covers all misclassified examples in the first and the second ranks (), almost of the third rank (), and some parts of the fourth rank (). It shows that the increase of covers more candidate classes, while the larger size of creates a higher risk to employ an unnecessarily large number of binary classifiers. On the other hand, if is too low, the target class may be removed. However, a suitable can be obtained by general tuning techniques. For our experiment, we just define to be for all of datasets without fine-tuning which is good enough to demonstrate the effectiveness of the VCF algorithm.
5 Experiments
In this section, we design the experimental setting to evaluate the performance of the proposed methods. We compare our methods with the traditional algorithms, i.e., the DDAG, the ADAG, and Max Wins. We divide this section into two parts as experimental protocols, and results & discussions.
5.1 Experimental Protocol
We run experiments on sixteen datasets from the UCI Machine Learning Repository Blake98 including Page Block, Glass, Segment, Arrhyth, Mfeat-factor, Mfeat-fourier, Mfeat-karhunen, Mfeat-zernike, Optdigit, Pendigit, Primary tumor, Libras Movement, Abalone, Krkopt, Spectrometer, and Letter (see Table 1). For the datasets containing both training data and test data, we added up both of them into one set, and used 5-fold cross validation for evaluating the classification accuracy.
| Datasets | #Cases | #Classes | #Features |
|---|---|---|---|
| Page Block | 5,473 | 5 | 10 |
| Glass | 214 | 6 | 9 |
| Segment | 2,310 | 7 | 18 |
| Arrhyth | 438 | 9 | 255 |
| Mfeat-factor | 2,000 | 10 | 216 |
| Mfeat-fourier | 2,000 | 10 | 76 |
| Mfeat-karhunen | 2,000 | 10 | 64 |
| Mfeat-zernike | 2,000 | 10 | 47 |
| Optdigit | 5,620 | 10 | 62 |
| Pendigit | 10,992 | 10 | 16 |
| Primary tumor | 315 | 13 | 15 |
| Libras Movement | 360 | 15 | 90 |
| Abalone | 4,098 | 16 | 8 |
| Krkopt | 28,056 | 18 | 6 |
| Spectrometer | 475 | 21 | 101 |
| Letter | 20,052 | 26 | 16 |
In these experiments, we scaled data to be in [-1,1] and employed two kernel functions i.e., the Polynomial kernel , and the RBF kernel . For the polynomial kernel we applied the same set of degrees to all datasets, and for the RBF kernel we applied the set of degrees to Page Block, Glass, Segment, Mfeat-zernike, Pendigit, Libras Movement, Abalone, Krkopt, and Letter, and applied the set of degrees to the other datasets. The default parameter of regularization parameter was used for model construction; this parameter is used to trade off between error of the SVM on training data and margin maximization. In the training phase, we used software package version 6.02 Joachims98 ; Joachims99 to create the binary classifiers. For the DDAG and the ADAG, we examined all possible orders of classes for datasets having not more than 8 classes, whereas we randomly selected 50,000 orders for datasets having more than 8 classes, and we then calculated the average of accuracy of these orders.
5.2 Results & Discussions
We compare the original methods with their enhanced techniques in three tasks including: (1) the ADAG with the RADAG, (2) the DDAG with two improved approaches, i.e., SE and WE, and (3) Max Wins with VCF.
We also selected the best techniques from (1) and (2), i.e., the RADAG and WE, respectively, and compared them with Max Wins as the state of the art technique. These comparison results are shown in Table 2 to Table 5. Moreover, paired comparison among all of three traditional methods (the DDAG, the ADAG, and Max Wins), and all proposed techniques (SE, the RADAG, WE, and VCF) are concluded in Table 6.
The best accuracy among these methods is represented in bold-face. In addition, we used the one-tailed paired t-test technique to analyze the significant difference between the accuracies of the traditional algorithms and the proposed algorithms. To estimate the difference between accuracies, we use a
-fold cross-validation method Mitchell97 .To indicate the level of the confidence interval using a one-tailed paired t-test in the Table
2 to Table 5, the symbol ‘’ and ‘’ are used to represent that the corresponding method has higher accuracy, and lower accuracy compared to a baseline method, respectively. The number of symbols shows the level of confidence interval for estimating the difference between accuracies of two algorithms i.e., one symbol, two symbols, and three symbols represent 90%, 95%, and 99% respectively.| Polynomial | RBF | |||
|---|---|---|---|---|
| Data sets | ADAG | RADAG | ADAG | RADAG |
| Page Block | 93.597 | 93.541 | 93.562 | 93.555 |
| Glass | 63.879 | 64.019 | 63.084 | 63.318 |
| Segment | 93.207 | 93.236 | 93.348 | 93.366 |
| Arrhyth | 63.489 | 63.470 | 58.049 | 57.991 |
| Mfeat-factor | 97.238 | 97.225 | 96.921 | 96.938 |
| Mfeat-fourier | 82.839 | 82.863 | 82.456 | 82.513 |
| Mfeat-karhunen | 96.864 | 96.863 | 96.890 | 96.900 |
| Mfeat-zernike | 82.368 | 82.413 | 81.867 | 81.888 |
| Optdigit | 98.995 | 98.999 | 98.620 | 98.630 |
| Pendigit | 99.400 | 99.402 | 99.313 | 99.320 |
| Primary tumor | 47.266 | 47.619 | 46.089 | 46.429 |
| Libras Movement | 73.218 | 73.194 | 72.289 | 72.569 |
| Abalone | 27.603 | 27.648 | 27.353 | 27.337 |
| Krkopt | 53.102 | 53.239 | 53.088 | 53.173 |
| Spectrometer | 54.445 | 54.842 | 50.808 | 51.579 |
| Letter | 88.668 | 88.787 | 89.989 | 90.090 |
The experimental results in Table 2 uses the ADAG as the baseline algorithm. It shows that the RADAG yields highest accuracy in several datasets. The results also show that, at % confidence interval, the RADAG performs statistically better than the ADAG in five datasets using the Polynomial kernel and better in three datasets using the RBF kernel. As shown in the table, the RADAG performs better when the number of classes is comparatively large, and does not perform well in the datasets with the small number of classes, i.e., the Page Block, and the Mfeat-factor with 5 and 10 classes, respectively. We believe that in case of datasets with the large number of classes, the variety of generalization errors of classifiers in consideration is rich and the RADAG is able to choose good classifiers freely, whereas the RADAG may be forced to select ineffective classifiers in case of the small number of classes, and it could lead to an incorrect output class.
| Polynomial | RBF | |||||
|---|---|---|---|---|---|---|
| Data sets | DDAG | SE | WE | DDAG | SE | WE |
| Page Block | 93.597 | 93.541 | 93.623 | 93.562 | 93.555 | 93.582 |
| Glass | 63.892 | 64.019 | 64.019 | 63.084 | 63.201 | 63.201 |
| Segment | 93.207 | 93.236 | 93.247 | 93.350 | 93.344 | 93.366 |
| Arrhyth | 63.490 | 63.527 | 63.527 | 58.048 | 57.991 | 58.162 |
| Mfeat-factor | 97.238 | 97.250 | 97.238 | 96.923 | 96.975 | 96.975 |
| Mfeat-fourier | 82.837 | 82.863 | 82.863 | 82.443 | 82.475 | 82.538 |
| Mfeat-karhunen | 96.863 | 96.875 | 96.875 | 96.861 | 96.850 | 96.988 |
| Mfeat-zernike | 82.362 | 82.400 | 82.350 | 81.869 | 81.888 | 81.863 |
| Optdigit | 98.994 | 99.013 | 99.008 | 98.618 | 98.643 | 98.630 |
| Pendigit | 99.399 | 99.404 | 99.402 | 99.312 | 99.318 | 99.320 |
| Primary tumor | 47.227 | 47.064 | 47.460 | 46.019 | 46.032 | 46.111 |
| Libras Movement | 73.142 | 73.264 | 73.472 | 72.283 | 72.569 | 72.431 |
| Abalone | 27.611 | 27.648 | 27.672 | 27.354 | 27.330 | 27.398 |
| Krkopt | 53.101 | 53.263 | 53.472 | 53.088 | 53.212 | 53.320 |
| Spectrometer | 54.373 | 54.632 | 54.421 | 50.821 | 51.316 | 51.842 |
| Letter | 88.609 | 88.707 | 88.835 | 89.903 | 89.977 | 90.294 |
Table 3 shows the experimental results of SE and WE compared with the DDAG as the baseline algorithm. Both WE and SE have higher accuracy than the traditional DDAG in almost all datasets. The results also show that at % confidence interval, SE performs statistically significantly better than the DDAG in four datasets using the Polynomial kernel and significantly better than the DDAG in two datasets using the RBF kernel. It is similar to the previous comparison between the ADAG and the RADAG that in datasets with the small number of classes, the classifier manipulation of SE may be forced to select inaccurate classifiers and it possibly leads to the misclassification. The results also show that WE performs statistically significantly better than the DDAG in five datasets in both cases of the Polynomial kernel and the RBF kernel. These results illustrate that WE can reduce the risk of selecting inaccurate classifiers compared to SE.
| Data sets | Polynomial | RBF | ||
|---|---|---|---|---|
| Max Wins | VCF | Max wins | VCF | |
| Page Block | 93.600 | 93.623 | 93.567 | 93.582 |
| Glass | 63.863 | 64.019 | 63.143 | 63.201 |
| Segment | 93.209 | 93.247 | 93.351 | 93.366 |
| Arrhyth | 63.489 | 63.527 | 58.048 | 58.162 |
| Mfeat-factor | 97.242 | 97.238 | 96.927 | 96.975 |
| Mfeat-fourier | 82.852 | 82.825 | 82.454 | 82.525 |
| Mfeat-karhunen | 96.879 | 96.875 | 96.952 | 96.963 |
| Mfeat-zernike | 82.338 | 82.350 | 81.825 | 81.863 |
| Optdigit | 99.004 | 99.013 | 98.631 | 98.630 |
| Pendigit | 99.402 | 99.402 | 99.315 | 99.320 |
| Primary tumor | 47.394 | 47.460 | 46.508 | 46.191 |
| Libras Movement | 73.194 | 73.472 | 72.373 | 72.431 |
| Abalone | 27.614 | 27.672 | 27.375 | 27.398 |
| Krkopt | 53.149 | 53.475 | 53.146 | 53.328 |
| Spectrometer | 54.263 | 54.421 | 51.026 | 51.842 |
| Letter | 88.706 | 88.869 | 90.112 | 90.316 |
| Polynomial | RBF | |||||
|---|---|---|---|---|---|---|
| Data sets | Max Wins | RADAG | WE | Max Wins | RADAG | WE |
| Page Block | 93.600 | 93.541 | 93.623 | 93.567 | 93.555 | 93.582 |
| Glass | 63.863 | 64.019 | 64.019 | 63.143 | 63.318 | 63.201 |
| Segment | 93.209 | 93.236 | 93.247 | 93.351 | 93.366 | 93.366 |
| Arrhyth | 63.489 | 63.470 | 63.527 | 58.048 | 57.991 | 58.162 |
| Mfeat-factor | 97.242 | 97.225 | 97.238 | 96.927 | 96.938 | 96.975 |
| Mfeat-fourier | 82.852 | 82.863 | 82.863 | 82.454 | 82.513 | 82.538 |
| Mfeat-karhunen | 96.879 | 96.863 | 96.875 | 96.952 | 96.900 | 96.988 |
| Mfeat-zernike | 82.338 | 82.413 | 82.350 | 81.825 | 81.888 | 81.863 |
| Optdigit | 99.004 | 98.999 | 99.008 | 98.631 | 98.630 | 98.630 |
| Pendigit | 99.402 | 99.402 | 99.402 | 99.315 | 99.320 | 99.320 |
| Primary tumor | 47.394 | 47.619 | 47.460 | 46.508 | 46.429 | 46.111 |
| Libras Movement | 73.194 | 73.194 | 73.472 | 72.373 | 72.569 | 72.431 |
| Abalone | 27.614 | 27.648 | 27.672 | 27.375 | 27.337 | 27.398 |
| Krkopt | 53.149 | 53.239 | 53.472 | 53.146 | 53.173 | 53.320 |
| Spectrometer | 54.263 | 54.842 | 54.421 | 51.026 | 51.579 | 51.842 |
| Letter | 88.706 | 88.787 | 88.835 | 90.112 | 90.090 | 90.294 |
We further analyze the results comparing WE and SE on the Letter dataset which consists of classes and 325 binary learners, as shown in Fig. 11. These 325 classifiers in the figure are sorted in ascending order by the generalization error, and this sequence of classifiers is maintained in the classification phase. SE requires classifiers and WE requires classifiers in this case, and the generalization error of the worst binary classifier in WE is almost five times lower than in SE (the largest generalization errors of all binary SVMs used in SE and WE are 0.015, 0.073, respectively). As a result, the average performance of the binary classifiers in WE is higher than SE.
As shown in Table 4 with Max Wins as the baseline method, VCF yields higher accuracy than Max Wins in almost all of datasets. The results show that, at % confidence interval, in the Polynomial kernel VCF performs statistically significantly better than Max Wins in four datasets, and in the RBF kernel VCF performs statistically significantly better than Max Wins in five datasets. The previous three tables show that our proposed methods improve the accuracy of the ADAG, the DDAG, and Max Wins significantly.
Next, we select the best algorithm in each table from the first two tables, i.e, the RADAG, and WE, and then compare them to Max Wins. According to experimental result in Table 5, at % confidence interval, the RADAG performs statistically significantly better than Max Wins in five datasets using the Polynomial kernel, and significantly higher than Max Wins in one dataset using the RBF kernel. In case of the small number of classes, it is possible that the RADAG will have the effect mentioned above. For WE, the results show that it performs statistically significantly better than Max Wins in four datasets in case of the Polynomial kernel and significantly better than Max Wins in five datasets in case of the RBF kernel. There is no any dataset in which Max Wins has significantly higher accuracy than WE.
| Traditional Methods | Proposed Methods | |||||||
|---|---|---|---|---|---|---|---|---|
| Kernel Function | Algorithms | DDAG | ADAG | Max Wins | SE | RADAG | WE | VCF |
| Polynomial | DDAG | 1-15-0 | 2-14-0 | 4-11-1 | 5-9-2 | 5-11-0 | 6-10-0 | |
| ADAG | 2-13-1 | 4-11-1 | 5-9-2 | 4-12-0 | 5-11-0 | |||
| Max Wins | 3-12-1 | 5-9-2 | 4-12-0 | 4-12-0 | ||||
| SE | 2-14-0 | 4-11-1 | 4-11-1 | |||||
| RADAG | 2-12-2 | 4-10-2 | ||||||
| WE | 1-15-0 | |||||||
| RBF | DDAG | 2-14-0 | 2-14-0 | 2-14-0 | 3-13-0 | 5-11-0 | 5-11-0 | |
| ADAG | 2-14-0 | 1-15-0 | 3-13-0 | 5-11-0 | 5-11-0 | |||
| Max Wins | 2-12-2 | 1-14-1 | 5-11-0 | 5-11-0 | ||||
| SE | 2-13-1 | 4-12-0 | 3-13-0 | |||||
| RADAG | 3-13-0 | 3-13-0 | ||||||
| WE | 1-15-0 | |||||||
Table 6 summarizes paired comparisons of all algorithms including the traditional techniques, and the proposed works based on both of the Polynomial kernel and the RBF kernel. We show the win-draw-loss record (s) of the algorithm in the column against the algorithm in the row. A win-draw-loss record reports how many datasets the method in the column is better than the method in the row (win), is equal (draw), or is worse (loss) at % confidence interval. As summarized in the table, our proposed methods are better than all previous works i.e., the DDAG, the ADAG, and Max Wins. WE and VCF give the highest accuracy among all of our methods. The result also shows that VCF gives a little better results compared to WE. However, as mentioned before in Section 4.3, the accuracies of VCF are the ones without fine-tuning, and higher accuracies can be expected if fine-tuning is performed to find the optimal for VCF.
5.3 Computational Time
The computational times of all methods are shown in Fig. 12 and Fig. 13. We can classify algorithms according to the time requirement into three groups, for an -class problem: 1) times i.e., the DDAG, the ADAG, SE, and the RADAG, 2) average about half of time of i.e., WE, 3) i.e., Max Wins, and VCF.
The results show that algorithms in the first and the second groups require comparatively low running time in all datasets, especially when the number of classes is relatively large, while the larger the number of classes, the more running time the algorithms in the third group requires. WE in the second group requires classifiers in the best case and classifiers in the worst case; however, in our experimental results WE takes approximately half of time required by the algorithms in the third group. For the RADAG, though the number of classes affects the running time for reordering process, it takes a little time even when there are many classes. The algorithms in the third group need comparisons for a problem with classes. VCF needs more time to choose the final class from the set of candidate classes which can be obtained by re-using the previous results of binary classification.
The DDAG reduces the number of comparisons down to . SE spends a little time more than the DDAG for sorting the classifiers in the training phase. By reducing the depth of the path, the ADAG and SE require comparisons of binary classifiers. WE consumes more time than SE due to each round of classification can reduce only one classifier while SE can eliminate all classifiers built from the discarded class. The number of testing classifiers for WE is equal to that for Max Wins in the worst case; fortunately, the experimental results show that WE actually spends only half of Max Wins’ times in the average case. The RADAG needs a little time more than the ADAG for reordering the order of classes. Note that, the minimum weight perfect matching algorithm, which is used in the reordering algorithm, runs in time bounded by Cook97 , where is the number of nodes (classes) in the graph and is the number of edges (binary classifiers). The RADAG will reorder the order of classes in every level, except for the last level. The order of classes in the top level is reordered only once and we use the order to evaluate every test example. Hence for classifying each test data, we need times of reordering, where each time the number of classes is reduced by half. Therefore, the running time of the RADAG is bounded by , where is much larger than .
6 Conclusion
Max Wins is a powerful combining technique with a need of number of classifications for an -class problem, while the DDAG and the ADAG reduce the number of classifications to . We study the characteristics of these previous methods that lead to wrong classification results. We believe that the performances of them depend on the BCRTs. In case of Max Wins, if there exists only one BCRT giving an incorrect answer, it may convey misclassification due to equal voting or another non-target class reaching the largest vote, while in cases of the DDAG and the ADAG, if only one of BCRTs in the sequence of selected classifiers makes a mistake, the whole system will give the wrong output. We investigate the well-organized combination of the binary models including BCRTs in classification process to provide a more precise final result.
In this research, we propose four methods for overcoming the above weakness of the previous works. All our proposed methods are based on the same principle that if the information about genearalization ability is accurately measured, then it is able to be employed for enhancing the performance of the classification. In this paper, the generalization performance is estimated by -fold cross-validation technique, and we show that it is more suitable than previously used measures in other frameworks, such as the margin size and the number of support vectors. Our proposed methods are the Reordering Adaptive Directed Acyclic Graph (RADAG), Strong Elimination of the classifiers (SE), Weak Elimination of the classifiers (WE), and Voting based Candidate (VCF). The RADAG is an enhanced version for the ADAG by using the minimum weight perfect matching for selecting the optimal pair of classes in each level with minimum generalization error. Compared to the ADAG, the RADAG is not only superior in terms of accuracy, but also maintains the same testing time (). Next, We propose two improved algorithms for the DDAG, i.e. SE and WE. In SE, a sequence of binary classifiers selected by minimum generalization error is applied to eliminate the candidate classes until only one class remained and assigned as the final output class. SE provides better accuracy than the DDAG. The testing time of SE is the same as the traditional DDAG and the RADAG, with a number of applied classifiers equal to . We also propose the other enhanced version for the DDAG, called WE. This approach aims to efficiently use as many as possible of the classifiers with low generalization errors. This is different from the process of the DDAG and SE in which all binary classifiers related to a defeated class are ignored when the defeated class is removed from the candidate classes. In WE, however, a classifier will be ignored only if all of two related classed of that classifier are discarded from the candidate output classes, and this process enables WE to efficiently employ good classifiers. WE gives significantly higher performance compared to the DDAG, and requires the number of classifications on average about half of the number of all possible binary classifiers.
Additionally, we propose VCF by applying the voting technique to carefully select the high competitive classes with high confidence. The remaining candidate classes are recursively eliminated by using WE. Although the number of classifications of VCF is equal to that of Max Wins, it shows the highest accuracy compared to all the other algorithms.
Finally, more experiments were conducted to compare our proposed algorithms and Max Wins in order to find the suitable scenario for using each of them. The RADAG should be chosen when the number of classes is large and the classification time is the most concern. VCF shows the highest accuracy among our proposed algorithms, and it should be selected when the time constraints is not the main concern. In a general case, WE is the most suitable method because it is superior to Max Wins in terms of accuracy and time. All of our techniques apply the generalization performance for organizing the use of the binary classifiers. This measure can be optimally estimated by the mechanism of
-fold cross-validation that is independent of base learners. Consequently, all our proposed methods can be also applied to other base classifiers such as logistic regression, perceptron, linear discriminant analysis, etc. The estimation of generalization errors using k-fold cross validation requires additional computation, and this can be thought of as a drawback of our methods. However, the estimation is done in the offline training phase, and thus it does not affect the performance in the classification phase.
7 Acknowledgment
The authors would like to thank Dr.Peerapon Vateekul for his valuable comments on an earlier version of this paper. This research is partially supported by the Thailand Research Fund, and the Graduate School, Chulalongkorn University.
References
-
(1)
V.Vapnik , Statistical Learning Theory, New York, Wiley, 1998.
-
(2)
V.Vapnik, An overview of statistical learning theory, IEEE Transactions on Neural Networks. 10 (1999) 988-999.
- (3) J.H.Friedman, Another approach to polychotomous classification, Technical report, Stanford University, Department of Statistics, 1996.
-
(4)
C.C.Chang, L.Chien, and Y.Lee, A novel framework for multi-class classification via ternary smooth support vector machine, Pattern Recognition 44 (2011) 1235-1244.
- (5) J.Manikandan, and B.Venkataramani, Study and evaluation of a multi-class SVM classifier using diminishing learning technique, Neurocomputing 73 (2010) 1676-1685.
-
(6)
T.G.Dietterich, and G.Bakiri, Solving multiclass learning problems via error-correcting output codes, Journal of Artificial Intelligence Research 2 (1995) 263-286.
- (7) E.L.Allwein, R.E.Schapire, and Y.Singer, Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers, Journal of Machine Learning Research 1 (2000) 113-141.
- (8) M.A.Bagheri, G.Montazer, and E.Kabir, A subspace approach to error correcting output codes, Pattern Recognition Letters 34 (2013) 176-184.
- (9) L.I.Kuncheva, Using diversity measures for generating error-correcting output codes in classifier ensembles, Pattern Recognition Letters 26 (2005) 83-90.
- (10) A.C.Lorena, and A.C.P.L.F.Carvalho Evaluation functions for the evolutionary design of multiclass Support Vector Macines, International Journal of Computational Intelligence and Applications 8 (2009) 53-68.
- (11) J.Platt, N.Cristianini, and J.Shawe-Taylor, Large margin DAGs for multiclass classification, Proceedings of Neural Information Processing Systems, MIT Press (2000) 547-553.
- (12) C.Hsu, and C.Lin, A comparison of methods for multiclass support vector machines, IEEE Transactions on Neural Networks 13 (2002) 415-425.
- (13) B.Kijsirikul,and N.Ussivakul, Multiclass support vector machines using adaptive directed acyclic graph., Proceedings of International Joint Conference on Neural Networks (IJCNN) (2002) 980-985.
- (14) F.Takahashi, and S.Abe, Optimizing directed acyclic graph support vector machines, Proceedings of Artificial Neural Networks in pattern recognition (2003) 166-170.
- (15) A.C.Lorena, and A.C.P.L.F.Carvalho, Building binary-tree-based multiclass classifiers using separability measures, Neurocomputing 73 (2010) 2837-2845.
- (16) R.Li, A.Li, T.Wang, and L.Li, Vector projection method for unclassifiable region of support vector machine, Expert Systems with Applications 38 (2011) 856-861.
- (17) C.Blake, E.Keogh, and C.Merz, UCI repository of machine learning databases, Department of Information and Computer Science, University of California, Irvine, 1998.
- (18) V.Vapnik, The Nature of Statistical Learning Theory, London, UK, Springer-Verlag, 1995.
- (19) V. N.Vapnik, and A.Y.Chervonenkis, Teoriya Raspoznavaniya Obrazov: Statisticheskie Problemy Obucheniya. (Russian) [Theory of Pattern Recognition: Statistical Problems of Learning]., Moscow: Nauka, 1974.
- (20) P.L.Bartlett, and J.Shawe-Taylor, Generalization performance of support vector machines and other pattern classifiers, Advances in Kernel Methods - Support Vector Learning, MIT Press, Cambridge, USA, (1999) 43-54.
- (21) C.Burges, A tutorial on support vector machines for pattern recognition, Data Mining and Knowledge Discovery 2 (1998) 121-167.
- (22) T.Mitchell, Machine Learning, McGraw Hill, 1997.
- (23) R.Johnson and, G.Bhattacharyya, Statistics: principles and methods, New York, Wiley, 2001.
- (24) W.Cook, and A.Rohe, Computing minimum-weight perfect matchings, Technical Report 97863, Forschungsinstitut fr Diskrete Mathematik, Universitt Bonn, (1999).
- (25) T.Joachims, Making large-scale SVM learning practical, Advances in Kernel Methods - Support Vector Learning, MIT Press (1998).
-
(26)
T.Joachims, SVM, http://ais.gmd.de/~thorsten/svm_light, (1999).