Object detection is one of several fundamental topics in computer vision. The task of object detection is to identify predefined objects in a given images using knowledge gained through analysis of a set of labelled positive and negative exemplars. Viola and Jones’ face detection algorithm forms the basis of many of the state-of-the-art real-time algorithms for object detection tasks.
The most commonly adopted evaluation method by which to compare the detection performance of different algorithms is the Receiver Operating Characteristic (ROC) curve. The curve illustrates the varying performance of a binary classifier system as its discrimination threshold is altered. In the face and human detection literature researchers are often interested in the low false positive area of the ROC curve since this region characterizes the performance needed for most real-world vision applications. This is due to the fact that object detection is a highly asymmetric classification problem as there are only ever a small number of target objects among the millions of background patches in a single test image. A false positive rate of per scanning window would result in thousands of false positives in a single image, which is impractical for most applications. For many tasks, and particularly human detection, researchers also report the partial area under the ROC curve (pAUC), typically over the range and false positives per image . As the name implies, pAUC is calculated as the area under the ROC curve between two specified false positive rates (FPRs). It summarizes the practical performance of a detector and often is the primary performance measure of interest.
Although pAUC is the metric of interest that has been used to evaluate detection performance, Most classifiers do not directly optimize this evaluation criterion, and as a result, often under-perform. In this paper, we present a principled approach for learning an ensemble classifier which directly optimizes the partial area under the ROC curve, where the range over which the area is calculated may be selected according to the desired application. Built upon the structured learning framework, we thus propose here a novel form of ensemble classifier which directly optimizes the partial AUC score, which we call pAUCEns. As with all other boosting algorithms, our approach learns a predictor by building an ensemble of weak classification rules in a greedy fashion. It also relies on a sample re-weighting mechanism to pass the information between each iteration. However, unlike traditional boosting, at each iteration, the proposed approach places a greater emphasis on samples which have the incorrect ordering111The positive sample has an incorrect ordering if it is ranked below the negative sample. In other words, we want all positive samples to be ranked above all negative samples. to achieve the optimal partial AUC score. The result is the ensemble learning method which yields the scoring function consistent with the correct relative ordering of positive and negative samples and optimizes the partial AUC score in a false positive rate range where .
(1) We propose a new ensemble learning approach which explicitly optimizes the partial area under the ROC curve (pAUC) between any two given false positive rates. The method is of particular interest in the wide variety of applications where performance is most important over a particular range within the ROC curve. The approach shares similarities with conventional boosting methods, but differs significantly in that the proposed method optimizes a multivariate performance measure using structured learning. Our design is simple and a conventional boosting-based visual detector can be transformed into a pAUCEns-based visual detector with very few modifications to the existing code. Our approach is efficient since it exploits both the efficient weak classifier training and the efficient cutting plane solver for optimizing the partial AUC score in the structural SVM setting. (2) We show that our approach is more intuitive and simpler to use than alternative algorithms, such as Asymmetric AdaBoost  and Cost-Sensitive AdaBoost , where one needs to cross-validate the asymmetric parameter from a fixed set of discrete points. Furthermore, it is unclear how one would set the asymmetric parameter in order to achieve a maximal pAUC score for a specified false positive range. To our knowledge, our approach is the first principled ensemble method that directly optimizes the partial AUC in an arbitrary false positive range . (3) Experimental results on several data sets, especially on challenging human detection data sets, demonstrate the effectiveness of the proposed approach. Our pedestrian detector performs better than or on par with the state-of-the-art, despite the fact that our detector only uses two standard low-level image features.
Various ensemble classifiers have been proposed in the literature. Of these AdaBoost is one the most well known as it has achieved tremendous success in computer vision and machine learning applications. In object detection, the cost of missing a true target is often higher than the cost of a false positive. Classifiers that are optimal under the symmetric cost, and thus treat false positives and negatives equally, cannot exploit this information. Several cost sensitive learning algorithms, where the classifier weights a positive class more heavily than a negative class, have thus been proposed.
Viola and Jones introduced the asymmetry property in Asymetric AdaBoost (AsymBoost) 
. However, the authors reported that this asymmetry is immediately absorbed by the first weak classifier. Heuristics are then used to avoid this problem. Peng proposed a fully-corrective asymmetric boosting method which does not have this problem. Note that one needs to carefully cross-validate the asymmetric parameter in order to achieve the desired result. Masnadi-Shirazi and Vasconcelos  proposed a cost-sensitive boosting algorithm based on the statistical interpretation of boosting. Their approach is to optimize the cost-sensitive loss by means of gradient descent. Shen proposed LACBoost and FisherBoost to address this asymmetry issue in cascade classifiers . Most works along this line address the pAUC evaluation criterion indirectly. In addition, one needs to carefully cross-validate the asymmetric parameter in order to maximize the detection rate in a particular false positive range.
Several algorithms that directly optimize the pAUC score have been proposed in bioinformatics [9, 11]. Komori and Eguchi optimize the pAUC using boosting-based algorithms . This algorithm is heuristic in nature. Narasimhan and Agarwal develop a structural SVM based method which directly optimizes the pAUC score 
. They demonstrate that their approach, which uses a support vector method, significantly outperforms several existing algorithms, including pAUCBoost and asymmetric SVM . Building on Narasimhan and Agarwal’s work, we propose the principled fully-corrective ensemble method which directly optimizes the pAUC evaluation criterion. The approach is flexible and can be applied to an arbitrary false positive range . To our knowledge, our approach is the first principled ensemble learning method that directly optimizes the partial AUC in a false positive range not bounded by zero. It is important to emphasize here the difference between our approach and that of .  train a linear structural SVM while our approach learns the ensemble of classifiers. For pedestrian detection, HOG with the ensemble of classifiers reduces the average miss-rate over HOG+SVM by more than .
Bold lower-case letters, , , denote column vectors and bold upper-case letters, , , denote matrices. Let be the set of positive training data and be the set of negative training data. A set of all training samples can be written as where and . We denote by a set of all possible outputs of weak learners. Assuming that we have possible weak learners, the output of weak learners for positive and negative data can be represented as where and , respectively. Here is the label predicted by the weak learner on the positive training data . Each column of the matrix represents the output of all weak learners when applied to the training instance . Each row of the matrix represents the output predicted by the weak learner on all the training data. The goal is to learn a set of binary weak learners and a scoring function, , that has good performance in terms of the pAUC between some specified false positive rates and where .
Structured learning approach for optimizing pAUC
Before we propose our approach, we briefly review the concept of SVM , in which our ensemble learning approach is built upon. Unless otherwise stated, we follow the symbols used in . The area under the empirical ROC curve (AUC) can be defined as,
where , , denotes the negative instance in ranked in the -th position amongst negative samples in descending order of scores. , and correspond to the sum of detection rates at , , and , respectively.
Given a training sample , our objective is to find a linear function that optimizes the pAUC in an FPR range of . We cast this pAUC optimization problem as a structural learning task. For any ordering of the training instances, the relative ordering of positive instances and negative instances is represented via a matrix where,
We define the correct relative ordering of as where . The pAUC loss in the false positive range of with respect to can be written as,
where denotes the index of the negative instance consistent with the matrix . We define the joint feature map of the form
The choice of over guarantees that the variable , which optimizes , will also produce the scoring function that achieves the optimal partial AUC score. The above problem can be summarized as the following convex optimization problem :
and . Note that denote the correct relative ordering and denote any arbitrary orderings.
2 Our approach
In order to design an ensemble-like algorithm for the pAUC, we first introduce a projection function, , which projects an instance vector to . This projection function is also known as the weak learner in boosting. In contrast to the previously described structured learning, we learn the scoring function, which optimizes the area under the curve between two false positive rates of the form: where is the linear coefficient vector and denote a set of binary weak learners. Let us assume that we have already learned a set of all projection functions. By using the same pAUC loss, , as in (1), and the same feature mapping, , as in (5), the optimization problem we want to solve is:
and . is the projected output for positive and negative training samples. where and it is defined as,
The only difference between (6) and (7) is that the original data is now projected to a new non-linear feature space. We will show how this can further improved the pAUC score in the experiment section. The dual problem of (7) can be written as (see supplementary),
where is the dual variable, denotes the dual variable associated with the inequality constraint for and . To derive the Lagrange dual problem, the following KKT condition is used,
Finding best weak learners
In this section, we show how one can explicitly learn the projection function, . We use the idea of column generation to derive an ensemble-like algorithm similar to LPBoost . The condition for applying the column generation is that the duality gap between the primal and dual problem is zero (strong duality). By inspecting the KKT condition, at optimality, (10) must hold for all . In other words, must hold for all .
For the weak learner in the current working set, the corresponding condition in (10) is satisfied by the current solution. For the weak learner that are not yet selected, they do not appear in the current restricted optimization problem and the corresponding . It is easy to see that if for any that are not in the current working set, then the current solution is already the globally optimal one. Hence the subproblem for selecting the best weak learner is:
In other words, we pick the weak learner with the value most deviated from zero. At iteration , we pick the most optimal weak learner from . Substituting (8) into (11), the subproblem for generating the optimal weak learner at iteration can be defined as,
where , , index the positive training samples (), the negative training samples () and the entire training samples (,,), respectively. Here
For decision stumps, the last equation in (2) is always valid since the weak learner set is negation-closed . In other words, if , then , and vice versa. Here . For decision stumps, one can flip the inequality sign such that and . In fact, any linear classifiers of the form are negation-closed. Using (2) to choose the best weak learner is not heuristic as the solution to (11) decreases the duality gap the most for the current solution. See supplementary for more details.
Optimizing weak learners’ coefficients
We solve for the optimal that minimizes our objective function (7). However, the optimization problem (7) has an exponential number of constraints, one for each matrix . As in [16, 10], we use the cutting plane method to solve this problem. The basic idea of the cutting plane is that a small subset of the constraints are sufficient to find an -approximate solution to the original problem. The algorithm starts with an empty constraint set and it adds the most violated constraint set at each iteration. The QP problem is solved using linear SVM and the process continues until no constraint is violated by more than
. Since, the quadratic program is of constant size and the cutting plane method converges in a constant number of iterations, the major bottleneck lies in the combinatorial optimization (over) associated with finding the most violated constraint set at each iteration. Narasimhan and Agarwal show how this combinatorial problem can be solved efficiently in a polynomial time . We briefly discuss their efficient algorithm in this section.
The combinatorial optimization problem associated with finding the most violated constraint can be written as,
The trick to speed up (14) is to note that any ordering of the instances that is consistent with yields the same objective value, in (15). In addition, one can break down (14) into smaller maximization problems by restricting the search space from to the set where
Here represents the set of all matrices in which the ordering of the scores of two negative instances, and , is consistent. The new optimization problem is now easier to solve as the set of negative instances over which the loss term in (15) is computed is the same for all orderings in the search space. This simplification allows one to reduce the computational complexity of (15) to . Interested reader may refer to .
Our final ensemble classifier has a similar form as the AdaBoost-based object detector of . Based on Algorithm LABEL:ALG:main, step ① and ② of our algorithm are exactly the same as . Similar to AdaBoost, in step ① plays the role of sample weights associated to each training sample. The major difference between AdaBoost and our approach is in step ③ and ④ where the weak learner’s coefficient is computed and the sample weights are updated. In AdaBoost, the weak learner’s coefficient is calculated as where and is the indicator function. The sample weights are updated with . We point this out here since a minimal modification is required in order to transform the existing implementation of AdaBoost to pAUCEns. Given the existing code of AdaBoost and the publicly available implementation of , our pAUCEns can be implemented in less than lines of codes. A computational complexity analysis of our approach can be found in the supplementary.
Synthetic data set
We first illustrate the effectiveness of our approach on a synthetic data set similar to the one used in . We compare pAUCEns against the baseline AdaBoost, Cost-Sensitive AdaBoost (CS-AdaBoost)  and Asymmetric AdaBoost (AsymBoost) . We use vertical and horizontal decision stumps as the weak classifier. We evaluate the partial AUC score of different algorithms at FPRs. For each algorithm, we train a strong classifier consisting of and weak classifiers. Additional details of the experimental set-up are provided in the supplementary. Fig. 1 illustrates the boundary decision222We set the threshold such that the false positive rate is . and the pAUC score. Our approach outperforms all other asymmetric classifiers. We observe that pAUCEns places more emphasis on positive samples than negative samples to ensure the highest detection rate at the left-most part of the ROC curve (FPR ). Even though we choose the asymmetric parameter, , from a large range of values, both CS-AdaBoost and AsymBoost perform slightly worse than our approach. AdaBoost performs worst on this toy data set since it optimizes the overall classification accuracy. However as the number of weak classifiers increases ( stumps), we observe all algorithms perform similarly on this simple toy data set. This observation could explain the success of AdaBoost in many object detection applications even though AdaBoost only minimizes the symmetric error rate.
In the next experiment, we train a strong classifier of weak classifiers and compare the performance of different classifiers at FPR of . We choose this value since it is the node learning goal often used in training a cascade classifier. Also we only learn weak classifiers since the first node of the cascade often contains a small number of weak classifiers for real-time performance. For pAUCEns, we set the value of to be . In Fig. 2, we display the decision boundary of each algorithm, and display both their pAUC score (in the FPR range ) and detection rate at false positive rate. We observe that our approach and AsymBoost have the highest detection rate at false positive rate. However, our approach outperforms AsymBoost on a pAUC score. We observe that our approach places more emphasis on positive samples near the corners (at , , and angles) than other algorithms.
Protein-protein interaction prediction
In this experiment, we compare our approach with existing algorithms which optimize pAUC in bioinformatics. The problem we consider here is a protein-protein interaction prediction , in which the task is to predict whether a pair of proteins interact or not. We used the data set labelled ‘Physical Interaction Task in Detailed feature type’, which is publicly available on the internet333http://www.cs.cmu.edu/~qyj/papers_sulp/proteins05_pages/feature-download.html. The data set contains protein pairs known to be interacting (positive) and a random set of protein pairs labelled as non-interacting (negative). We use a subset of features as in . We randomly split the data into two groups: for training/validation and for evaluation. We choose the best regularization parameter form , , , , by -fold cross validation. We repeat our experiments times using the same regularization parameter. We train a linear classifier as our weak learner using LIBLINEAR . We set the maximum number of boosting iterations to and report the pAUC score of our approach in Table 1. Baselines include SVM, SVM, pAUCBoost and Asymmetric SVM. Our approach outperforms all existing algorithms which optimize either AUC or pAUC . We attribute our improvement over SVM , as a result of introducing a non-linearity into the original problem. This phenomenon has also been observed in face detection as reported in .
Comparison to other asymmetric boosting
Here we compare pAUCEns against several boosting algorithms previously proposed for the problem of object detection, namely, AdaBoost with Fisher LDA post-processing , AsymBoost  and CS-AdaBoost . The results of AdaBoost are also presented as the baseline. For each algorithm, we train a strong classifier consisting of weak classifiers. We then calculate the pAUC score by varying the threshold value in the FPR range . For each algorithm, the experiment is repeated times and the average pAUC score is reported. For AsymBoost, we choose from , , , by cross-validation. For CS-AdaBoost, we choose from , , , by cross-validation. We evaluate the performance of all algorithms on
vision data sets: USPS digits, scenes and face data sets. See supplementary for more details on feature extraction. We report the experimental results in Table2. From the table, pAUCEns demonstrates the best performance on all three vision data sets.
|Ada + LDA||()||()||()|
Average pAUC scores and their standard deviations on vision data sets at various boosting iterations. All experiments are repeatedtimes. The best average performance is shown in boldface.
Pedestrian detection - Strong classifier
We evaluate our approach on the pedestrian detection task. We train our approach on the INRIA pedestrian data set. For the positive training data, we use all INRIA cropped pedestrian images. To generate the negative training data, we first train the cascade classifier with nodes using Viola and Jones’ approach. We then combine random negative windows generated in the first node with another negative windows generated in the subsequent nodes. The resulting negative windows are used for training the strong classifier. We generate a large pool of features by combining the histogram of oriented gradient (HOG) features  and covariance (COV) features444Covariance features capture the relationship between different image statistics and have been shown to perform well in our previous experiments. However, other discriminative features can also be used here instead, , Haar-like features, Local Binary Pattern (LBP) , Sketch Tokens  and self-similarity of low-level features (CSS) . . Additional details of HOG and COV parameters are provided in the supplementary. We use weighted linear discriminant analysis (WLDA) as weak classifiers. We train weak classifiers and set multi-exits . To be more specific, we set the threshold at , , , and weak classifiers. These exits reduce the evaluation time during testing significantly. The regularization parameter is cross-validated from . Since we have not carefully cross-validated a finer range of , tuning this parameter could yield a further improvement. The training time of our approach is under two hours on a parallelized quad core Xeon machine.
During evaluation, each test image is scanned with
pixels step stride and the scale ratio of input image pyramid is. The overlapped detection windows are merged using the greedy non-maximum suppression strategy as introduced in . We use the continuous AUC evaluation software of Sermanet  and report the pAUC score between FPPI ( false positive), FPPI ( false positives), FPPI ( false positives) and FPPI ( false positives) in Table 3. From the table, we observe that setting the value of to be minimal () yeilds the best pAUC score at FPPI . As we increase the FPPI range, the higher value of tends to perform better. This table clearly illustrates the advantage of our approach.
Fig. 3 compares the performance of our approach with other state-of-the-art algorithms on the INRIA pedestrian data set. We use the evaluation software of Dollár , which computes the AUC from discrete points sampled between FPPI . Our approach performs second best on this data set. It performs comparable to VeryFast  which trains multiple detectors at multiple scale. Upon a closer observation, our pAUCEns performs slightly better than VeryFast when the number of FPPI is less then and VeryFast performs slightly better when the number of FPPI is greater . We evaluate our strong classifier on TUD-Brussels and ETH pedestrian data sets but we observe that the detection results contain a large number of false positives. Instead of bootstrapping with more negative samples as in [6, 24], we train a cascade classifier in the next section.
Pedestrian detection - Cascade classifier
In this section, we train a cascade classifier using our pAUCEns. We train our detector on INRIA training set and evaluate the detector on INRIA, TUD-Brussels and ETH test sets. On both TUD-Brussels and ETH data sets, we upsample the original image to pixels before applying our pedestrian detector. We train the human detector with a combination of HOG and COV features as previously described. To achieve the node learning goal of the cascade (each node achieves an extremely high detection rate () and a moderate false positive rage ()), we optimize the pAUC in the FPR range . We train a multi-exit cascade  with exit. In this experiment, we use the software of  to compute the continuous AUC score in the FPPI range . We sort different algorithms based on the pAUC score in the FPPI range and report the results in Fig. 4. We compare our proposed approach with the baseline HOGCOV classifier (using AdaBoost). We observe that our approach reduces the average miss-rate over HOGCOV by on INRIA test set. From Fig. 4, our approach achieves similar performance to the state-of-the-art detector. We then break-down experimental results of different measures using the partial AUC score (FPPI range ) in Table 4. On average, our approach performs best on the large evaluation setting where pedestrians are at least pixels tall. On other settings, our approach yields competitive results to the state-of-the-art detector in that category. In summary, our approach performs better than or on par with the state-of-the-art despite its simplicity (in comparison to LatSvm — a part-based approach which models unknown parts as latent variables). In addition, the current detector is only trained with two discriminative visual features (HOG and COV). Applying additional discriminative features, , LBP  or motion features , could further improve the overall detection performance.
|Reasonable (min. 50 pixels tall & no/partial occlusion) - Partial AUC(0,0.1)%|
|Large (min. 100 pixels tall) - Partial AUC(0,0.1)%|
|Near (min. 80 pixels tall) - Partial AUC(0,0.1)%|
|Medium (min. 30 pixels tall and max. 80 pixels tall) - Partial AUC(0,0.1)%|
We have proposed a new ensemble learning method for object detection. The proposed approach is based on optimizing the partial AUC score in the FPR range . Extensive experiments demonstrate the effectiveness of the proposed approach in visual detection tasks. We plan to explore the possibility of applying the proposed approach to the multiple scales detector of  in order to improve the detection results of very low resolution pedestrian images.
-  R. Benenson, M. Mathias, R. Timofte, and L. V. Gool. Pedestrian detection at 100 frames per second. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2012.
-  R. Benenson, M. Mathias, T. Tuytelaars, and L. V. Gool. Seeking the strongest rigid detector. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2013.
-  N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., volume 1, 2005.
-  A. Demiriz, K. Bennett, and J. Shawe-Taylor. Linear programming boosting via column generation. Mach. Learn., 46(1–3):225–254, 2002.
L. E. Dodd and M. S. Pepe.
Partial auc estimation and regression.Biometrics, 59(3):614–623, 2003.
-  P. Dollár, Z. Tu, P. Perona, and S. Belongie. Integral channel features. In Proc. of British Mach. Vis. Conf., 2009.
-  P. Dollár, C. Wojek, B. Schiele, and P. Perona. Pedestrian detection: An evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell., 34(4):743–761, 2012.
-  R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. LIBLINEAR: A library for large linear classification. J. Mach. Learn. Res., 9:1871–1874, 2008.
-  M.-J. Hsu and H.-M. Hsueh. The linear combinations of biomarkers which maximize the partial area under the roc curves. Comp. Stats., 28(2):1–20, 2012.
-  T. Joachims, T. Finley, and C.-N. J. Yu. Cutting-plane training of structural svms. Mach. Learn., 77(1):27–59, 2009.
-  O. Komori and S. Eguchi. A boosting method for maximizing the partial area under the roc curve. BMC Bioinformatics, 11(1):314, 2010.
O. Komori and S. Eguchi.
Boosting learning algorithm for pattern recognition and beyond.IEICE Trans. Infor. and Syst., 94(10):1863–1869, 2011.
-  J. J. Lim, C. L. Zitnick, and P. Dollar. Sketch Tokens: A learned mid-level representation for contour and object detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2013.
-  H. Masnadi-Shirazi and N. Vasconcelos. Cost-sensitive boosting. IEEE Trans. Pattern Anal. Mach. Intell., 33(2):294–309, 2011.
-  Y. Mu, S. Yan, Y. Liu, T. Huang, and B. Zhou. Discriminative local binary patterns for human detection in personal album. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., Anchorage, AK, US, 2008.
-  H. Narasimhan and S. Agarwal. A structural svm based approach for optimizing partial auc. In Proc. Int. Conf. Mach. Learn., 2013.
-  M.-T. Pham, V.-D. D. Hoang, and T.-J. Cham. Detection with multi-exit asymmetric boosting. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2008.
-  Y. Qi, Z. Bar-Joseph, and J. Klein-Seetharaman. Evaluation of different biological data and computational classification methods for use in protein interaction prediction. Proteins: Struct., Func., and Bioinfor., 63(3):490–500, 2006.
-  P. Sermanet, K. Kavukcuoglu, S. Chintala, and Y. LeCun. Pedestrian detection with unsupervised multi-stage feature learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2013.
-  C. Shen, P. Wang, S. Paisitkriangkrai, and A. van den Hengel. Training effective node classifiers for cascade classification. Int. J. Computer Vision, 103(3):326–347, 2013.
-  O. Tuzel, F. Porikli, and P. Meer. Pedestrian detection via classification on Riemannian manifolds. IEEE Trans. Pattern Anal. Mach. Intell., 30(10):1713–1727, 2008.
-  P. Viola and M. Jones. Fast and robust classification using asymmetric AdaBoost and a detector cascade. In Proc. Adv. Neural Inf. Process. Syst., pages 1311–1318. MIT Press, 2002.
-  P. Viola and M. J. Jones. Robust real-time face detection. Int. J. Comp. Vis., 57(2):137–154, 2004.
-  S. Walk, N. Majer, K. Schindler, and B. Schiele. New features and insights for pedestrian detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., San Francisco, US, 2010.
P. Wang, C. Shen, N. Barnes, and H. Zheng.
Fast and robust object detection using asymmetric totally-corrective
IEEE Trans. Neural Networks and Learning Systems, 23(1):33–46, 2012.
-  X. Wang, T. X. Han, and S. Yan. An HOG-LBP human detector with partial occlusion handling. In Proc. IEEE Int. Conf. Comp. Vis., 2009.
-  J. Wu, S. C. Brubaker, M. D. Mullin, and J. M. Rehg. Fast asymmetric learning for cascade face detection. IEEE Trans. Pattern Anal. Mach. Intell., 30(3):369–382, 2008.
S.-H. Wu, K.-P. Lin, C.-M. Chen, and M.-S. Chen.
Asymmetric support vector machines: low false-positive learning under the user tolerance.In Proc. of Intl. Conf. on Knowledge Discovery and Data Mining, 2008.