1 Introduction
The support vector machine (SVM) Vapnik98 ; Vapnik99
is a high performance learning algorithm constructing a hyperplane to separate twoclass data by maximizing the margin between them. There are two approaches for extending SVMs to multiclass problems, i.e., solving the problem by formulating all classes of data under a single optimization, and combining several twoclass subproblems. However, the difficulty and complexity to solve the problem with the first method are due to the increase of the number of classes and the size of training data, so the second method is more suitable for practical use. In this paper, we focus on the second approach.
For constructing a multiclass classifier from binary ones, the method called oneagainstone trains each binary classifier on only two out of classes, and builds possible classifiers. Several strategies have been proposed for combining the trained classifiers to make the final classification for an unseen data. Friedman Friedman96 suggested the combination strategy called Max Wins. In the classification process of Max Wins, every binary classifier provides one vote for its preferred class and the class with the largest vote will be set to be the final output. Chang and Lee Chang11 investigated an adaptive framework to manage a nuisance vote which is a vote for an unrelated class by allowing a classifier to make a nonvote for data of unrelated class. Instead of a binary classifier, they employed a ternary classifier that consists of two particular classes and the rest of the classes fused as the third class for solving this problem.
Vapnik Vapnik98 proposed the oneagainsttherest approach working by constructing a set of binary classifiers in which each classifier is learned from all examples in the class, and the remaining classes labeled with the positive and negative classes, respectively. The class corresponding to the classifier with the highest output value is used to make the final output. Moreover, Manikandan and Venkataramani Manikandan10 adapted the traditional oneagainsttherest to work as a sequential classifier. All classifiers will be ordered corresponding to their misclassification. This method needs a lower number of classifiers on avearge compared with the traditional oneagainsttherest, but both algorithms have the same problem in the training phase because of the difficulty for calculating the absolutely separating hyperplane between a class and all of the other classes.
Dietterich and Bakiri Dietterich95 introduced the Error Correcting Output Code (ECOC) based on the fundamental of information theory. For a given code matrix with rows and columns, each element contains either 1’, or 1’. Each column denotes the bit string showing the combination of positive and negative classes for constructing a binary classifier, and each row of the code matrix indicates the unique bit string for representing a specific class (each bit string is called a codeword). Allwein et al. Allwein2000 extended the coding method by adding the third symbol 0’ as “ don’t care bit” to allow the binary model learned without considering some particular classes. Unlike the previous method, the number of classes for training a binary classifier can be varied from 2 to classes. Based on these two systems for an classes problem, the maximum numbers of different binary classifiers are Dietterich95 , and Bagheri2013 , respectively. Design of code matrices with different subsets of binary classifiers gives different abilities for separating classes, and the problem of selecting a suitable subset of binary classifiers is complicated with a large size of
. To obtain the suitable code matrix, some techniques using the Genetic Algorithm have been proposed
Kuncheva2005 ; Lorena2009 . In the classification phase, a test example is classified by all classifiers corresponding to the column of the code matrix, and then the class with the closet codeword is assigned to the final output class.Platt et al. Platt99 proposed the Decision Directed Acyclic Graph (DDAG) in order to reduce evaluation time Hsu02 . In each round, a binary model will be randomly selected from all classifiers. The binary classification result is employed to eliminate the candidate output classes, and to ignore all binary classifiers related to the defeated class. It guarantees that the number of classifications (applied classifiers) of the DDAG is always . This recursive task will be applied until there is only one last candidate class. However, the misclassification of the DDAG can be occurred at the time when selected binary classifiers related to the target class (hence forth BCRT) give the wrong answer. The more times the number of BCRTs are applied, the more chance the misclassification is produced by the DDAG. In order to reduce this risk, Kijsirikul and Ussivakul Kijsirihul02 proposed the Adaptive Directed Acyclic Graphs (ADAG) that has a reversed triangular structure of the DDAG. It requires only times or less that the target class is tested against the other classes, while the DDAG possibly requires at most times.
In addition, there have been many attempts that apply some information such as the margin size Platt99 , the number of support vectors Abe03 , and the separability measures between classes Lorena11 ; Li11 , to improve the performance of the multiclass classification. The margin size and the number fo support vectors were applied for selecting the suitable twoclass classifiers in the DDAG Platt99 ; Abe03 . The separability measure was employed for automatically constructing a binary tree of multiclass classification based on the concept of the minimum spanning tree Lorena11 . Li, et al. Li11 used similar information to vote the preferred class for data in unclassifiable region for both the oneagainstone and the oneagainsttherest techniques.
In this research, we investigate the framework for enhancing three wellknown methods, which are the DDAG, the ADAG, and Max Wins. Max Wins is currently recognized as thestateoftheart combining algorithm and it is also the most powerful technique among all of our focused works with a need of number of classifications for an class problem, while the other two approaches reduce the number of classifications to . We study the characteristics of these methods that lead to wrong classification results. The first two techniques have the same hierarchical structure and have the same weak point that they “trust on individual opinion” for making decision to discard the candidate classes. Intuitively, if only one of BCRTs makes a mistake, the whole system will give the wrong output. The last technique as the high performance one, Max Wins is based on the concept of “trust on most popular opinion” for making decision to select the output class. If all of BCRTs give the correct answer, Max Wins will always provide the correct output class. However, if there exists only one of BCRTs give the wrong answer, it may lead to misclassification due to equal voting, or other nontarget classes reaching the largest vote as shown later in the paper. Examples which are incorrectly classified in this scenario can be recovered by our proposed strategies.
In this paper, we demonstrate that the above traditional methods can be improved based on the same idea that if we access further important information of generalization performance
of all binary classifiers and properly estimate it, it can be employed for enhancing the performance of the methods. Based on this idea, we propose four novel approaches including (1) the Reordering Adaptive Directed Acyclic Graph (RADAG), (2) Strong Elimination of the classifiers (SE), (3) Weak Elimination of the classifiers (WE), and (4) Voting based Candidate Filtering (VCF). The first approach, the next two approaches, and the last approach are improved from the ADAG, the DDAG, and Max Wins, respectively. We also empirically evaluate our methods by comparing them with the traditional methods on the sixteen datasets from the UCI Machine Learning Repository
Blake98 .This paper is organized as follows. Section 2 reviews the traditional multiclass classification frameworks. Section 3 describes how to properly estimate the generalization performance of binary classifiers. Section 4 presents our proposed methodologies. Section 5 performs experiments and explains the results and discussions. Section 6 concludes the research.
2 Multiclass Support Vector Machines
2.1 Max Wins
For an class problem, all possible pairs of twoclass data are learned for constructing classifiers. All binary classifiers are applied for voting the preferred class. A class with maximum vote will be assigned as the final output class. This method is called Max Wins Friedman96 . However, in case that there exists more than one class giving the same maximum vote, the final output class can be obtained by random selection from candidate classes with the equal maximumvote. An example of the classification using this technique for a fourclass problem is shown in Fig. 1. Each class will be voted (solidline) or ignored (dashline) by all related binary models. For example, class has three related classifiers, i.e., 1 vs 2, 1 vs 3, and 1 vs 4. The voting result of class , class , class and class are three, zero, one, and, two, respectively. In this case, class has the largest score, and therefore it is assigned as the final output class.
2.2 Decision Directed Acyclic Graphs
Platt et al. Platt99 introduced a learning algorithm using the Directed Acyclic Graph (DAG) to represent the classification task, called the Decision Directed Acyclic Graph (DDAG). This architecture represents a set of nodes connected by edges with no cycles. Each edge has an orientation and each node has either 0 or 2 edges. Among these nodes, there exists a root node which is the unique node with no edge pointing into it. In a DDAG, the nodes are arranged with a triangular shape in which each node is labeled with an element of a boolean function. There exists a single root node at the top, two nodes in the second layer, and so on until the final layer of leaves for an class problem.
To make a classification, an example with an unknown class label is evaluated by the nodes as binary decision functions. The binary output result in each layer is applied to eliminate the candidate output classes and the binary classifiers related to the defeated class are removed. At the first layer (see Fig. 2), the root node can be randomly selected from all possible classifiers and there are candidate output classes. After the root node is tested, its binary result is employed to eliminate the candidate output classes and the binary classifiers corresponding to the defeated class are discarded. In the next layer, the remaining binary classifiers are randomly selected to continue the same process in which some classes are eliminated from the remaining candidate classes. The process is repeated until there is only one class remained which is then assigned as the final output class. This algorithm requires only decision nodes in order to obtain the final answer.
One disadvantage of the DDAG is that its classification result is affected by the sequence of binary classifiers randomly selected in the evaluation process. Platt et al. also proposed the other method that prefers the binary decision function with the higher generalization performance measured by its margin sizes, called the large margin DAGs Platt99 . The margin size () is a parameter for bounding the generalization ability of the binary SVM as shown in terms of the VCdimension in Eq. (2). It illustrates that the generalization performance of the binary model is proportional to the size of the margin. A binary classifier with the larger margin size will be firstly applied in each round of the evaluation step. Moreover, Takahashi and Abe Abe03 proposed a similar framework that employed the number of support vectors as a performance measure. In this method, the generalization error () for classes and was bounded by Eq. (1) Vapnik95 :
(1) 
where is the number of support vectors for classes and and is the number of training data for classes and .
2.3 Adaptive Directed Acyclic Graphs
In the DDAG, binary classification result of a previously employed binary classifier is used to eliminate a candidate output classe, and there are only current remaining candidate classes that can be possibly assigned as the final output class. Therefore, the misclassification of a selected BCRT is the crucial point.
The ADAG was originally designed to reduce this risk of the DDAG by using reversed triangular structure Kijsirihul02 . In an class problem, there are nodes at the top, nodes in the second layer and so on until the lowest layer of the final node, as illustrated in Fig. 3. Like the DDAG, binary output results of the ADAG in each layer are applied to discard candidate output classes and the binary classifiers related to the defeated classes are also ignored. Therefore, the ADAG also evaluates only nodes to obtain the final answer.
According to the critical issue of misclassification mentioned above, even only one selected classifier related to the target class provides a wrong answer, the misclassification on the final output class cannot be avoided. Hence, the number of times the target class is tested against other classes indicates the risk of misclassification. The DDAG requires at most times that the target class is tested against other classes, while the ADAG requires only times or less. This shows that the opportunities of the target class tested against other classes on the ADAG is much lower than the DDAG.
3 An Estimation of the Generalization Performances of Binary Support Vector Machines
The generalization performance of a learning model is the actual performance evaluated on unseen data. For support vector machines, a model is trained by using the concept of the Structure Risk Minimization principle Vapnik74 in which the generalization performance of the model is estimated based on both terms of the complexity of model (the VC dimension of approximating functions) and the quality of fitting training data (empirical error). Consider the problem of binary classification where dataset of samples in real dimensional space is randomly independent identically distributed observations drawn according to . The expected risk (
) with probability at least
can be bounded by the following equation Bartlett99 ; Burges98 :(2) 
where there is a corresponding constant
for all probability distributions,
is the number of labeled examples in with margin less than , indicates the radius of the smallest sphere that contains all the data points, and is the distance between the hyperplane and the closest points of the training set (margin size). The first and second terms of inequality in Eq. (2) denote the bound of the empirical error, and the VC dimension, respectively.In our frameworks, the generalization ability will be applied to improve the multiclass classification. Although there have been many attempts to use some performance measures such as the margin size Platt99 , the number of support vectors Abe03 , they may not accurately reflect the actual performance of each binary SVM. Consider a twoclass problem where hyperplanes and are learning models created to separate the positive and the negative examples. Suppose that they provide different margin sizes of and , and the different numbers, and , of labeled examples in with the margin less than their margin sizes, respectively. In case that the parameters and are fixed, there are only two parameters including and that affect the performance of the learning model (as the parameters and , as well as and are the same for the same pair of a twoclass problem). Now consider twolearning models learned from different pairs of a twoclass problem. In case that the parameters , and are fixed, according to inequality in Eq. (2), obviously, if we use only , , or combination of them, they are not sufficient to represent the whole term of their generalization abilities. This shows that a binary model with the larger margin size may not provide more accurate result of classification. The use of only the number of support vectors is also shown in Burges98 that it is not predictive for generalization ability.
As described above, the generalization ability can be employed to enhance the performance of multiclass classification, by carefully design algorithms which utilize this information as a selection measure for good classifiers. We believe that the generalization performance of binary SVMs can be directly estimated by fold crossvalidation Mitchell97 (see Algorithm 1), and it can be used to fairly compare the performances of binary SVMs on different twoclass problems. Below we give an example which demonstrates that fold crossvalidation is more suitable for estimating the generalizaiton performance of the classifiers than the other measures used by the previous methods, i.e. the number of support vectors, the margin size.
Fig. 4 shows the generalization performance measured by the previous methods Platt99 ; Abe03 , and fold crossvalidation, which we propose to use as the performance measure, for the Letter dataset with 26 classes, by applying the polynomial kernel of . Fig. 4 (a) illustrates that the trend of estimated generalization error by fold crossvalidation is very closed to the actual risk, while the other two techniques give high variation. To further investigation in more details, we select about of all classifiers to show in Fig. 4 (bd); these figures illustrate the comparisons between the actual risk and the estimated generalization errors with different measures, i.e., CV Bound, SV Bound, and Normalized Margin Bound, respectively. In each figure, classifiers are sorted in ascending order by the estimated generalization errors. It is expected that if a specific measure is a good estimator for generalization error, its value should be in the same trend as the actual risk (its value shoud increase with the increase of the actual risk). A good trend is found in Fig. 4 (b), while the other two methods give no clear trend and contain confusing patterns. In order to evaluate the efficiency of these estimating methods, we apply the correlation analysis between two variables Johnson2001 , i.e., the actual risks and these three estimated generalization errors. These evaluations are based on 325 classifiers as in Fig. 4 (a), and the statistical rvalues of them are 0.805, 0.372, and 0.230 as shown in Fig. 4 (bd), respectively. The rvalues also confirm that CV Bound and actual risk have high correlation, while the other two methods give very low correlation. They show that fold crossvalidation is more suitable to be the measure for the performance of binary classifiers. According to the above reason, we apply this measure in our research.
4 The proposed methods
The combination of binary SVMs with high generalization performance directly affects the accuracy of the multiclass classification. In this section, we introduce four enhanced approaches based on the previous techniques i.e., the ADAG, the DDAG, and Max Wins by applying the generalization abilities in order to select suitable binary classifiers. An improvement of the ADAG is called the Reordering Adaptive Directed Acyclic Graph (RADAG). There are two improved versions for the DDAG i.e., Strong Elimination of the classifiers (SE) and Weak Elimination of the classifiers (WE). The last technique is Votingbased Candidate Filtering (VCF) enhanced from Max Wins. To increase the classification accuracy, the generalization estimated by fold crossvalidation is utilized as the goodness measure of classifiers in our frameworks.
4.1 Reordering Adaptive Directed Acyclic Graph
The ADAG is designed to reduce the number of times the binary classifiers related to the target class are applied, from at most times required by the DDAG, to times or less. However, binary classifiers in the first level of the ADAG are still randomly selected, and its misclassification can be produced at the time even when only one BCRT gives a wrong answer. In this section, we introduce a more effective method which uses the minimum weight perfect matching to select the optimal pairs of classes in each level with minimum generalization error. We called the method the Reordering Adaptive Directed Acyclic Graph (RADAG).
The structure of the RADAG is similar to the ADAG, but they are different in the initialization of the binary classifiers in the top level and the order of classes in lower levels (see Fig. 4.1). The reordering algorithm with minimum weight perfect matching is described in Algorithm 2. The algorithm selects the optimal order of classes in each level. It is different from the ADAG in that the initial order of classes in the ADAG is obtained randomly, and the matching of classes in successive levels depends on the classification results of nodes from the previous level. For the RADAG, the reordering process will be applied to the remaining candidate classes in all levels for determining the optimal sequence of them.
To select the optimal set of classifiers, the generalization measure in Section 3 is used as a criterion. This scheme provides less chance to predict the wrong class from all possible orders. Among classifiers, classifiers which have the smallest sum of generalization errors will be used in the classification.
Let be a graph with node set and edge set . Each node in denotes one class and each edge indicates one binary classifier of which generalization error is estimated from Section 3 (see Fig. 6(a)). The output of the reordering algorithm for graph is a subset of edges with the minimum sum of generalization errors of all edges and each node in is met by exactly one edge in the subset (see Fig. 6(b)).
Given a real weight being generalization error for each edge of , the problem of reordering algorithm can be solved by the minimum weight perfect matching Cook97 that finds a perfect matching of minimum weight .
For , let .
is the set of edges with both endpoints in . The set of edges incident to
node in the nodeedge incidence matrix is denoted by .
The convex hull
of perfect matchings on a graph with even is given by
a)
b) for
c)
for all odd sets
with or by (a),(b) andd) for all odd sets with
where , and () means that is (is not) in the matching.
Hence, the minimum weight of a perfect matching is at least as large as the value of
(3) 
where satisfies “(a), (b), and (c)” or “(a), (b) and, (d)”. Therefore, the reordering problem can be solved by the integer program in Eq. (3).
4.2 Strong & Weak Elimination of Classifiers for Enhancing Decision Directed Acyclic Graph
According to the characteristic of the DDAG, binary classification results of the previously employed binary classifiers are used to eliminate the candidate output classes, and thus the final output class will be assigned with one of the remaining candidate classes. By using the random technique for selecting a binary classifier, the DDAG produces misclassification at the time when a BCRT with very low performance is selected and provides the wrong answer, as the target class will be discarded from the remaining candidate classes, and it is not possible to reach the correct output class. In this section, we propose the framework to enhance the performance of the DDAG to select the binary classifier with high performance based on the generalization abilities of binary classifiers as described in Section 3.
We propose two methods that are Strong Elimination of the classifiers (SE) and Weak Elimination of the classifiers (WE). Both algorithms are described in Algorithm 3 and Algorithm 4. We also show a classification process of SE and WE for an class problem in Fig. 7 and Fig. 8, respectively.
For both of the DDAG and SE, in each round, a defeated class will be removed from candidate output classes, and all binary classifiers related to the defeated class are ignored. Due to this reason, they guarantee number of classifications for an class problem. However, these ignored classifiers may have high generalization abilities and thus are helpful to eliminate the other remaining candidate classes. Therefore, we then propose WE to make use of binary classifiers with high generalization abilities.
According to the classifier elimination of WE, the number of classifications is bounded with the best case of , and the worst case of . However, WE provides the opportunities to employ better classifiers as shown in the Fig. 8. At round , suppose that classifier vs has lower generalization error than classifier vs , and both of them are active classifiers. In this case, it is possible that classifier vs can remove the class from the list of two remaining candidate classes, and can avoid using classifier vs with lower reliability that is unavoidable for SE as shown in Fig. 7.
4.3 Voting Based Candidate Filtering
Max Wins is one of high performance techniques that work based on the concept of “trust on the most popular opinion” for making decision to select the output class. If all of BCRTs give the correct answer, Max Wins will always provide the correct output class. It does not depend on the answers of the other binary classifiers. However, if only one of BCRTs gives a wrong answer, it may lead to misclassification due to equal voting, or another nontarget class reaching the largest vote. Fig. 9 shows an example of such cases, taken from our experiment on the Letter dataset (see Section for more details); Fig. 9(a) and (b) show the cases of equal voting and another nontarget class having the largest vote, respectively.
We propose a novel multiclass classification approach that alleviates the above problem of Max Wins, and uses the same concept “trust on the most popular opinion” for filtering out the low competitive classes. On the other hand, high competitive classes will be voted to be candidate output classes, though there exist some BCRTs providing the wrong answer. If there is more than one remaining class, the output class will be selected via the mechanism of WE. Our proposed technique aims to combine the strong point of both Max Wins and WE, and is called Voting based Candidate Filtering (VCF). The details of our algorithm are shown in Algorithm 5.
Let , and indicate the maximum of scores for all classes, and the score of class for a test data, respectively. Also let denotes the percentage of the difference between and . An example of the calculation of is shown in Fig. 9 (a), where E’, the score of class E’ = 23 points, and the score of class C’ = 24 points (as the top score). Then value can be calculated by . We also define to be the threshold of for considering class as a candidate for the target class; class will be accepted into the set of high competitive candidate classes if and only if its is less than or equal to . We want to keep the size of the filtered candidate classes as small as possible while still containing the target class.
A case study of high risk of misclassification in the Letter dataset including 4,010 examples where Max Wins provides 3,549 examples with the correct result, and 461 examples with high risk of misclassification. By a highrisk example, we mean (1) the example with an equal vote (the score of the target class is equal to those of other nontarget classes) and (2) the example with a vote less than the maximum vote that is then misclassified by Max Wins. These high risk examples will be hopefully recovered with the correct class label by our proposed algorithm. In our experiment, the highrisk examples includes 24 examples (around 5) with an equal vote, and 437 examples (around 95) with a vote less than the maximum as shown in Fig. 10, where represents the percentage of the difference between and the score of the target class. For each example, we calculate the rank of the voting score of the target class compared to the other nontarget classes, and consider only the first eight ranks. There are 24 examples (around ) in the first rank, while in the second to the eighth ranks, the numbers of examples are 171, 77, 31, 30, 15, 14, and 10 (around , and ), respectively. The examples with the different ranks have different ranges of values, such as, in the second rank, the values are varied from to , in the third rank, the values are varied from to , in the fourth rank, the values are varied from to , and so on.
According to this case study, there can be at most of examples that will be correctly classified with the correct class label by random selection of Max Wins, while the other of examples will be absolutely misclassified. We want to recover an example that is not correctly classified by Max Wins, as its actual target class is not in the first rank or its target class has equal vote with some other output classes. If is set as in the VCF algorithm, it will guarantee that all highrisk misclassified examples with values no greater than can be filtered into the set of the candidate output classes; in this case only the examples in the first rank ( of examples) will be selected. When we apply a bigger threshold, e.g. , it covers all misclassified examples in the first and the second ranks (), almost of the third rank (), and some parts of the fourth rank (). It shows that the increase of covers more candidate classes, while the larger size of creates a higher risk to employ an unnecessarily large number of binary classifiers. On the other hand, if is too low, the target class may be removed. However, a suitable can be obtained by general tuning techniques. For our experiment, we just define to be for all of datasets without finetuning which is good enough to demonstrate the effectiveness of the VCF algorithm.
5 Experiments
In this section, we design the experimental setting to evaluate the performance of the proposed methods. We compare our methods with the traditional algorithms, i.e., the DDAG, the ADAG, and Max Wins. We divide this section into two parts as experimental protocols, and results & discussions.
5.1 Experimental Protocol
We run experiments on sixteen datasets from the UCI Machine Learning Repository Blake98 including Page Block, Glass, Segment, Arrhyth, Mfeatfactor, Mfeatfourier, Mfeatkarhunen, Mfeatzernike, Optdigit, Pendigit, Primary tumor, Libras Movement, Abalone, Krkopt, Spectrometer, and Letter (see Table 1). For the datasets containing both training data and test data, we added up both of them into one set, and used 5fold cross validation for evaluating the classification accuracy.
Datasets  #Cases  #Classes  #Features 

Page Block  5,473  5  10 
Glass  214  6  9 
Segment  2,310  7  18 
Arrhyth  438  9  255 
Mfeatfactor  2,000  10  216 
Mfeatfourier  2,000  10  76 
Mfeatkarhunen  2,000  10  64 
Mfeatzernike  2,000  10  47 
Optdigit  5,620  10  62 
Pendigit  10,992  10  16 
Primary tumor  315  13  15 
Libras Movement  360  15  90 
Abalone  4,098  16  8 
Krkopt  28,056  18  6 
Spectrometer  475  21  101 
Letter  20,052  26  16 
In these experiments, we scaled data to be in [1,1] and employed two kernel functions i.e., the Polynomial kernel , and the RBF kernel . For the polynomial kernel we applied the same set of degrees to all datasets, and for the RBF kernel we applied the set of degrees to Page Block, Glass, Segment, Mfeatzernike, Pendigit, Libras Movement, Abalone, Krkopt, and Letter, and applied the set of degrees to the other datasets. The default parameter of regularization parameter was used for model construction; this parameter is used to trade off between error of the SVM on training data and margin maximization. In the training phase, we used software package version 6.02 Joachims98 ; Joachims99 to create the binary classifiers. For the DDAG and the ADAG, we examined all possible orders of classes for datasets having not more than 8 classes, whereas we randomly selected 50,000 orders for datasets having more than 8 classes, and we then calculated the average of accuracy of these orders.
5.2 Results & Discussions
We compare the original methods with their enhanced techniques in three tasks including: (1) the ADAG with the RADAG, (2) the DDAG with two improved approaches, i.e., SE and WE, and (3) Max Wins with VCF.
We also selected the best techniques from (1) and (2), i.e., the RADAG and WE, respectively, and compared them with Max Wins as the state of the art technique. These comparison results are shown in Table 2 to Table 5. Moreover, paired comparison among all of three traditional methods (the DDAG, the ADAG, and Max Wins), and all proposed techniques (SE, the RADAG, WE, and VCF) are concluded in Table 6.
The best accuracy among these methods is represented in boldface. In addition, we used the onetailed paired ttest technique to analyze the significant difference between the accuracies of the traditional algorithms and the proposed algorithms. To estimate the difference between accuracies, we use a
fold crossvalidation method Mitchell97 .To indicate the level of the confidence interval using a onetailed paired ttest in the Table
2 to Table 5, the symbol ‘’ and ‘’ are used to represent that the corresponding method has higher accuracy, and lower accuracy compared to a baseline method, respectively. The number of symbols shows the level of confidence interval for estimating the difference between accuracies of two algorithms i.e., one symbol, two symbols, and three symbols represent 90%, 95%, and 99% respectively.Polynomial  RBF  

Data sets  ADAG  RADAG  ADAG  RADAG 
Page Block  93.597  93.541  93.562  93.555 
Glass  63.879  64.019  63.084  63.318 
Segment  93.207  93.236  93.348  93.366 
Arrhyth  63.489  63.470  58.049  57.991 
Mfeatfactor  97.238  97.225  96.921  96.938 
Mfeatfourier  82.839  82.863  82.456  82.513 
Mfeatkarhunen  96.864  96.863  96.890  96.900 
Mfeatzernike  82.368  82.413  81.867  81.888 
Optdigit  98.995  98.999  98.620  98.630 
Pendigit  99.400  99.402  99.313  99.320 
Primary tumor  47.266  47.619  46.089  46.429 
Libras Movement  73.218  73.194  72.289  72.569 
Abalone  27.603  27.648  27.353  27.337 
Krkopt  53.102  53.239  53.088  53.173 
Spectrometer  54.445  54.842  50.808  51.579 
Letter  88.668  88.787  89.989  90.090 
The experimental results in Table 2 uses the ADAG as the baseline algorithm. It shows that the RADAG yields highest accuracy in several datasets. The results also show that, at % confidence interval, the RADAG performs statistically better than the ADAG in five datasets using the Polynomial kernel and better in three datasets using the RBF kernel. As shown in the table, the RADAG performs better when the number of classes is comparatively large, and does not perform well in the datasets with the small number of classes, i.e., the Page Block, and the Mfeatfactor with 5 and 10 classes, respectively. We believe that in case of datasets with the large number of classes, the variety of generalization errors of classifiers in consideration is rich and the RADAG is able to choose good classifiers freely, whereas the RADAG may be forced to select ineffective classifiers in case of the small number of classes, and it could lead to an incorrect output class.
Polynomial  RBF  

Data sets  DDAG  SE  WE  DDAG  SE  WE 
Page Block  93.597  93.541  93.623  93.562  93.555  93.582 
Glass  63.892  64.019  64.019  63.084  63.201  63.201 
Segment  93.207  93.236  93.247  93.350  93.344  93.366 
Arrhyth  63.490  63.527  63.527  58.048  57.991  58.162 
Mfeatfactor  97.238  97.250  97.238  96.923  96.975  96.975 
Mfeatfourier  82.837  82.863  82.863  82.443  82.475  82.538 
Mfeatkarhunen  96.863  96.875  96.875  96.861  96.850  96.988 
Mfeatzernike  82.362  82.400  82.350  81.869  81.888  81.863 
Optdigit  98.994  99.013  99.008  98.618  98.643  98.630 
Pendigit  99.399  99.404  99.402  99.312  99.318  99.320 
Primary tumor  47.227  47.064  47.460  46.019  46.032  46.111 
Libras Movement  73.142  73.264  73.472  72.283  72.569  72.431 
Abalone  27.611  27.648  27.672  27.354  27.330  27.398 
Krkopt  53.101  53.263  53.472  53.088  53.212  53.320 
Spectrometer  54.373  54.632  54.421  50.821  51.316  51.842 
Letter  88.609  88.707  88.835  89.903  89.977  90.294 
Table 3 shows the experimental results of SE and WE compared with the DDAG as the baseline algorithm. Both WE and SE have higher accuracy than the traditional DDAG in almost all datasets. The results also show that at % confidence interval, SE performs statistically significantly better than the DDAG in four datasets using the Polynomial kernel and significantly better than the DDAG in two datasets using the RBF kernel. It is similar to the previous comparison between the ADAG and the RADAG that in datasets with the small number of classes, the classifier manipulation of SE may be forced to select inaccurate classifiers and it possibly leads to the misclassification. The results also show that WE performs statistically significantly better than the DDAG in five datasets in both cases of the Polynomial kernel and the RBF kernel. These results illustrate that WE can reduce the risk of selecting inaccurate classifiers compared to SE.
Data sets  Polynomial  RBF  

Max Wins  VCF  Max wins  VCF  
Page Block  93.600  93.623  93.567  93.582 
Glass  63.863  64.019  63.143  63.201 
Segment  93.209  93.247  93.351  93.366 
Arrhyth  63.489  63.527  58.048  58.162 
Mfeatfactor  97.242  97.238  96.927  96.975 
Mfeatfourier  82.852  82.825  82.454  82.525 
Mfeatkarhunen  96.879  96.875  96.952  96.963 
Mfeatzernike  82.338  82.350  81.825  81.863 
Optdigit  99.004  99.013  98.631  98.630 
Pendigit  99.402  99.402  99.315  99.320 
Primary tumor  47.394  47.460  46.508  46.191 
Libras Movement  73.194  73.472  72.373  72.431 
Abalone  27.614  27.672  27.375  27.398 
Krkopt  53.149  53.475  53.146  53.328 
Spectrometer  54.263  54.421  51.026  51.842 
Letter  88.706  88.869  90.112  90.316 
Polynomial  RBF  

Data sets  Max Wins  RADAG  WE  Max Wins  RADAG  WE 
Page Block  93.600  93.541  93.623  93.567  93.555  93.582 
Glass  63.863  64.019  64.019  63.143  63.318  63.201 
Segment  93.209  93.236  93.247  93.351  93.366  93.366 
Arrhyth  63.489  63.470  63.527  58.048  57.991  58.162 
Mfeatfactor  97.242  97.225  97.238  96.927  96.938  96.975 
Mfeatfourier  82.852  82.863  82.863  82.454  82.513  82.538 
Mfeatkarhunen  96.879  96.863  96.875  96.952  96.900  96.988 
Mfeatzernike  82.338  82.413  82.350  81.825  81.888  81.863 
Optdigit  99.004  98.999  99.008  98.631  98.630  98.630 
Pendigit  99.402  99.402  99.402  99.315  99.320  99.320 
Primary tumor  47.394  47.619  47.460  46.508  46.429  46.111 
Libras Movement  73.194  73.194  73.472  72.373  72.569  72.431 
Abalone  27.614  27.648  27.672  27.375  27.337  27.398 
Krkopt  53.149  53.239  53.472  53.146  53.173  53.320 
Spectrometer  54.263  54.842  54.421  51.026  51.579  51.842 
Letter  88.706  88.787  88.835  90.112  90.090  90.294 
We further analyze the results comparing WE and SE on the Letter dataset which consists of classes and 325 binary learners, as shown in Fig. 11. These 325 classifiers in the figure are sorted in ascending order by the generalization error, and this sequence of classifiers is maintained in the classification phase. SE requires classifiers and WE requires classifiers in this case, and the generalization error of the worst binary classifier in WE is almost five times lower than in SE (the largest generalization errors of all binary SVMs used in SE and WE are 0.015, 0.073, respectively). As a result, the average performance of the binary classifiers in WE is higher than SE.
As shown in Table 4 with Max Wins as the baseline method, VCF yields higher accuracy than Max Wins in almost all of datasets. The results show that, at % confidence interval, in the Polynomial kernel VCF performs statistically significantly better than Max Wins in four datasets, and in the RBF kernel VCF performs statistically significantly better than Max Wins in five datasets. The previous three tables show that our proposed methods improve the accuracy of the ADAG, the DDAG, and Max Wins significantly.
Next, we select the best algorithm in each table from the first two tables, i.e, the RADAG, and WE, and then compare them to Max Wins. According to experimental result in Table 5, at % confidence interval, the RADAG performs statistically significantly better than Max Wins in five datasets using the Polynomial kernel, and significantly higher than Max Wins in one dataset using the RBF kernel. In case of the small number of classes, it is possible that the RADAG will have the effect mentioned above. For WE, the results show that it performs statistically significantly better than Max Wins in four datasets in case of the Polynomial kernel and significantly better than Max Wins in five datasets in case of the RBF kernel. There is no any dataset in which Max Wins has significantly higher accuracy than WE.
Traditional Methods  Proposed Methods  

Kernel Function  Algorithms  DDAG  ADAG  Max Wins  SE  RADAG  WE  VCF 
Polynomial  DDAG  1150  2140  4111  592  5110  6100  
ADAG  2131  4111  592  4120  5110  
Max Wins  3121  592  4120  4120  
SE  2140  4111  4111  
RADAG  2122  4102  
WE  1150  
RBF  DDAG  2140  2140  2140  3130  5110  5110  
ADAG  2140  1150  3130  5110  5110  
Max Wins  2122  1141  5110  5110  
SE  2131  4120  3130  
RADAG  3130  3130  
WE  1150 
Table 6 summarizes paired comparisons of all algorithms including the traditional techniques, and the proposed works based on both of the Polynomial kernel and the RBF kernel. We show the windrawloss record (s) of the algorithm in the column against the algorithm in the row. A windrawloss record reports how many datasets the method in the column is better than the method in the row (win), is equal (draw), or is worse (loss) at % confidence interval. As summarized in the table, our proposed methods are better than all previous works i.e., the DDAG, the ADAG, and Max Wins. WE and VCF give the highest accuracy among all of our methods. The result also shows that VCF gives a little better results compared to WE. However, as mentioned before in Section 4.3, the accuracies of VCF are the ones without finetuning, and higher accuracies can be expected if finetuning is performed to find the optimal for VCF.
5.3 Computational Time
The computational times of all methods are shown in Fig. 12 and Fig. 13. We can classify algorithms according to the time requirement into three groups, for an class problem: 1) times i.e., the DDAG, the ADAG, SE, and the RADAG, 2) average about half of time of i.e., WE, 3) i.e., Max Wins, and VCF.
The results show that algorithms in the first and the second groups require comparatively low running time in all datasets, especially when the number of classes is relatively large, while the larger the number of classes, the more running time the algorithms in the third group requires. WE in the second group requires classifiers in the best case and classifiers in the worst case; however, in our experimental results WE takes approximately half of time required by the algorithms in the third group. For the RADAG, though the number of classes affects the running time for reordering process, it takes a little time even when there are many classes. The algorithms in the third group need comparisons for a problem with classes. VCF needs more time to choose the final class from the set of candidate classes which can be obtained by reusing the previous results of binary classification.
The DDAG reduces the number of comparisons down to . SE spends a little time more than the DDAG for sorting the classifiers in the training phase. By reducing the depth of the path, the ADAG and SE require comparisons of binary classifiers. WE consumes more time than SE due to each round of classification can reduce only one classifier while SE can eliminate all classifiers built from the discarded class. The number of testing classifiers for WE is equal to that for Max Wins in the worst case; fortunately, the experimental results show that WE actually spends only half of Max Wins’ times in the average case. The RADAG needs a little time more than the ADAG for reordering the order of classes. Note that, the minimum weight perfect matching algorithm, which is used in the reordering algorithm, runs in time bounded by Cook97 , where is the number of nodes (classes) in the graph and is the number of edges (binary classifiers). The RADAG will reorder the order of classes in every level, except for the last level. The order of classes in the top level is reordered only once and we use the order to evaluate every test example. Hence for classifying each test data, we need times of reordering, where each time the number of classes is reduced by half. Therefore, the running time of the RADAG is bounded by , where is much larger than .
6 Conclusion
Max Wins is a powerful combining technique with a need of number of classifications for an class problem, while the DDAG and the ADAG reduce the number of classifications to . We study the characteristics of these previous methods that lead to wrong classification results. We believe that the performances of them depend on the BCRTs. In case of Max Wins, if there exists only one BCRT giving an incorrect answer, it may convey misclassification due to equal voting or another nontarget class reaching the largest vote, while in cases of the DDAG and the ADAG, if only one of BCRTs in the sequence of selected classifiers makes a mistake, the whole system will give the wrong output. We investigate the wellorganized combination of the binary models including BCRTs in classification process to provide a more precise final result.
In this research, we propose four methods for overcoming the above weakness of the previous works. All our proposed methods are based on the same principle that if the information about genearalization ability is accurately measured, then it is able to be employed for enhancing the performance of the classification. In this paper, the generalization performance is estimated by fold crossvalidation technique, and we show that it is more suitable than previously used measures in other frameworks, such as the margin size and the number of support vectors. Our proposed methods are the Reordering Adaptive Directed Acyclic Graph (RADAG), Strong Elimination of the classifiers (SE), Weak Elimination of the classifiers (WE), and Voting based Candidate (VCF). The RADAG is an enhanced version for the ADAG by using the minimum weight perfect matching for selecting the optimal pair of classes in each level with minimum generalization error. Compared to the ADAG, the RADAG is not only superior in terms of accuracy, but also maintains the same testing time (). Next, We propose two improved algorithms for the DDAG, i.e. SE and WE. In SE, a sequence of binary classifiers selected by minimum generalization error is applied to eliminate the candidate classes until only one class remained and assigned as the final output class. SE provides better accuracy than the DDAG. The testing time of SE is the same as the traditional DDAG and the RADAG, with a number of applied classifiers equal to . We also propose the other enhanced version for the DDAG, called WE. This approach aims to efficiently use as many as possible of the classifiers with low generalization errors. This is different from the process of the DDAG and SE in which all binary classifiers related to a defeated class are ignored when the defeated class is removed from the candidate classes. In WE, however, a classifier will be ignored only if all of two related classed of that classifier are discarded from the candidate output classes, and this process enables WE to efficiently employ good classifiers. WE gives significantly higher performance compared to the DDAG, and requires the number of classifications on average about half of the number of all possible binary classifiers.
Additionally, we propose VCF by applying the voting technique to carefully select the high competitive classes with high confidence. The remaining candidate classes are recursively eliminated by using WE. Although the number of classifications of VCF is equal to that of Max Wins, it shows the highest accuracy compared to all the other algorithms.
Finally, more experiments were conducted to compare our proposed algorithms and Max Wins in order to find the suitable scenario for using each of them. The RADAG should be chosen when the number of classes is large and the classification time is the most concern. VCF shows the highest accuracy among our proposed algorithms, and it should be selected when the time constraints is not the main concern. In a general case, WE is the most suitable method because it is superior to Max Wins in terms of accuracy and time. All of our techniques apply the generalization performance for organizing the use of the binary classifiers. This measure can be optimally estimated by the mechanism of
fold crossvalidation that is independent of base learners. Consequently, all our proposed methods can be also applied to other base classifiers such as logistic regression, perceptron, linear discriminant analysis, etc. The estimation of generalization errors using kfold cross validation requires additional computation, and this can be thought of as a drawback of our methods. However, the estimation is done in the offline training phase, and thus it does not affect the performance in the classification phase.
7 Acknowledgment
The authors would like to thank Dr.Peerapon Vateekul for his valuable comments on an earlier version of this paper. This research is partially supported by the Thailand Research Fund, and the Graduate School, Chulalongkorn University.
References

(1)
V.Vapnik , Statistical Learning Theory, New York, Wiley, 1998.

(2)
V.Vapnik, An overview of statistical learning theory, IEEE Transactions on Neural Networks. 10 (1999) 988999.
 (3) J.H.Friedman, Another approach to polychotomous classification, Technical report, Stanford University, Department of Statistics, 1996.

(4)
C.C.Chang, L.Chien, and Y.Lee, A novel framework for multiclass classification via ternary smooth support vector machine, Pattern Recognition 44 (2011) 12351244.
 (5) J.Manikandan, and B.Venkataramani, Study and evaluation of a multiclass SVM classifier using diminishing learning technique, Neurocomputing 73 (2010) 16761685.

(6)
T.G.Dietterich, and G.Bakiri, Solving multiclass learning problems via errorcorrecting output codes, Journal of Artificial Intelligence Research 2 (1995) 263286.
 (7) E.L.Allwein, R.E.Schapire, and Y.Singer, Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers, Journal of Machine Learning Research 1 (2000) 113141.
 (8) M.A.Bagheri, G.Montazer, and E.Kabir, A subspace approach to error correcting output codes, Pattern Recognition Letters 34 (2013) 176184.
 (9) L.I.Kuncheva, Using diversity measures for generating errorcorrecting output codes in classifier ensembles, Pattern Recognition Letters 26 (2005) 8390.
 (10) A.C.Lorena, and A.C.P.L.F.Carvalho Evaluation functions for the evolutionary design of multiclass Support Vector Macines, International Journal of Computational Intelligence and Applications 8 (2009) 5368.
 (11) J.Platt, N.Cristianini, and J.ShaweTaylor, Large margin DAGs for multiclass classification, Proceedings of Neural Information Processing Systems, MIT Press (2000) 547553.
 (12) C.Hsu, and C.Lin, A comparison of methods for multiclass support vector machines, IEEE Transactions on Neural Networks 13 (2002) 415425.
 (13) B.Kijsirikul,and N.Ussivakul, Multiclass support vector machines using adaptive directed acyclic graph., Proceedings of International Joint Conference on Neural Networks (IJCNN) (2002) 980985.
 (14) F.Takahashi, and S.Abe, Optimizing directed acyclic graph support vector machines, Proceedings of Artificial Neural Networks in pattern recognition (2003) 166170.
 (15) A.C.Lorena, and A.C.P.L.F.Carvalho, Building binarytreebased multiclass classifiers using separability measures, Neurocomputing 73 (2010) 28372845.
 (16) R.Li, A.Li, T.Wang, and L.Li, Vector projection method for unclassifiable region of support vector machine, Expert Systems with Applications 38 (2011) 856861.
 (17) C.Blake, E.Keogh, and C.Merz, UCI repository of machine learning databases, Department of Information and Computer Science, University of California, Irvine, 1998.
 (18) V.Vapnik, The Nature of Statistical Learning Theory, London, UK, SpringerVerlag, 1995.
 (19) V. N.Vapnik, and A.Y.Chervonenkis, Teoriya Raspoznavaniya Obrazov: Statisticheskie Problemy Obucheniya. (Russian) [Theory of Pattern Recognition: Statistical Problems of Learning]., Moscow: Nauka, 1974.
 (20) P.L.Bartlett, and J.ShaweTaylor, Generalization performance of support vector machines and other pattern classifiers, Advances in Kernel Methods  Support Vector Learning, MIT Press, Cambridge, USA, (1999) 4354.
 (21) C.Burges, A tutorial on support vector machines for pattern recognition, Data Mining and Knowledge Discovery 2 (1998) 121167.
 (22) T.Mitchell, Machine Learning, McGraw Hill, 1997.
 (23) R.Johnson and, G.Bhattacharyya, Statistics: principles and methods, New York, Wiley, 2001.
 (24) W.Cook, and A.Rohe, Computing minimumweight perfect matchings, Technical Report 97863, Forschungsinstitut fr Diskrete Mathematik, Universitt Bonn, (1999).
 (25) T.Joachims, Making largescale SVM learning practical, Advances in Kernel Methods  Support Vector Learning, MIT Press (1998).

(26)
T.Joachims, SVM, http://ais.gmd.de/~thorsten/svm_light, (1999).
Comments
There are no comments yet.