In a broad range of classification and decision making problems, one is given the advice or predictions of several classifiers, of unknown reliability, over multiple questions or queries. This scenario is different from the standard supervised setting, where each classifier accuracy can be assessed using available labeled data, and raises two questions: given only the predictions of several classifiers over a large set of unlabeled test data, is it possible to a) reliably rank them; and b) construct a meta-classifier more accurate than most classifiers in the ensemble?
Here we present a novel spectral approach to address these questions. First, assuming conditional independence between classifiers, we show that the off-diagonal entries of their covariance matrix correspond to a rank-one matrix. Moreover, the classifiers can be ranked using the leading eigenvector of this covariance matrix, as its entries are proportional to their balanced accuracies. Second, via a linear approximation to the maximum likelihood estimator, we derive the Spectral Meta-Learner (SML), a novel ensemble classifier whose weights are equal to this eigenvector entries. On both simulated and real data, SML typically achieves a higher accuracy than most classifiers in the ensemble and can provide a better starting point than majority voting, for estimating the maximum likelihood solution. Furthermore, SML is robust to the presence of small malicious groups of classifiers designed to veer the ensemble prediction away from the (unknown) ground truth.
Everyday, multiple decisions are made based on input and suggestions from several sources, either algorithms or advisers, of unknown reliability.
Investment companies handle their portfolios by combining reports from several analysts, each providing recommendations on buying, selling or holding multiple stocks [1, 2].
Central banks combine surveys of several professional forecasters to monitor rates of inflation, real GDP growth and unemployment [3, 4, 5, 6].
Biologists study the genomic binding locations of proteins, by combining or ranking the predictions of several peak detection algorithms applied to large-scale genomics data .
Physician tumor boards convene a number of experts from different disciplines to discuss patients whose diseases pose diagnostic and therapeutic challenges .
Peer review panels discuss multiple grant applications and make recommendations to fund or reject them .
The examples above describe scenarios in which several human advisers or algorithms, provide their predictions or answers to a list of queries or questions. A key challenge is to improve decision-making by combining these multiple predictions, of unknown reliability. Automating this process, of combining multiple predictors, is an active field of research in decision science (cci.mit.edu/research), medicine , business (,  and www.kaggle.com/competitions) and government (www.iarpa.gov/Programs/ia/ACE/ace.html and www.goodjudgmentproject.com ), as well as in statistics and machine learning.
), as well as in statistics and machine learning.
Such scenarios, whereby advisers of unknown reliability provide potentially conflicting opinions, or propose to take opposite actions, raise several interesting questions: How should the decision-maker proceed to identify who, among the advisers, is the most reliable? Moreover, is it possible for the decision-maker to cleverly combine the collection of answers from all the advisers and provide even more accurate answers?
In statistical terms, the first question corresponds to the problem of estimating prediction performances of pre-constructed classifiers (e.g., the advisers) in absence of class labels. Namely, each classifier was constructed independently on a potentially different training dataset (e.g., each adviser trained on his/her own using possibly different sources of information), yet they are all being applied to the same new test data (e.g., list of queries) for which labels are not available, either because they are expensive to obtain, or because they will only be available in the future, after the decision has been made. In addition, the accuracy of each classifier on its own training data is unknown. This scenario is markedly different from the standard supervised setting in machine learning and statistics. There, classifiers are typically trained on the same labeled data, and can be ranked, for example, by comparing their empirical accuracy on a common labeled validation set. In this paper we show that under standard assumptions of independence between classifier errors, their unknown performances can still be ranked even in the absence of labeled data.
The second question raised above corresponds to the problem of combining predictions of pre-constructed classifiers to form a meta-classifier with improved prediction performance. This problem arises in many fields, including combination of forecasts in decision science and crowdsourcing in machine learning, which have each derived different approaches to address it. If we had external knowledge or historical data to assess the reliability of the available classifiers we could use well-established solutions relying on panels of experts or forecast combinations [11, 12, 13, 14]. In our problem such knowledge is not always available and thus these solutions are in general not applicable. The oldest solution that does not require additional information is majority voting, whereby the predicted class label is determined by a rule of majority, with all advisers assigned the same weight.
More recently, iterative likelihood maximization procedures, pioneered by Dawid and Skene , have been proposed, in particular in crowdsourcing applications [16, 17, 18, 19, 20, 21, 22, 23]. Due to the non-convexity of the likelihood function, these techniques often converge only to a local, rather than global, maximum and require careful initialization. Furthermore, there are typically no guarantees on the quality of the resulting solution.
In this paper we address these questions via a novel spectral analysis, that yields four major insights:
Under standard assumptions of independence between classifier errors, in the limit of an infinite test set, the off-diagonal entries of the population covariance matrix of the classifiers correspond to a rank-one matrix.
The entries of the leading eigenvector of this rank-one matrix are proportional to the balanced accuracies of the classifiers. Thus, a spectral decomposition of this rank-one matrix provides a computationally efficient approach to rank the performances of an ensemble of classifiers.
A linear approximation of the maximum likelihood estimator yields an ensemble learner whose weights are proportional to the entries of this eigenvector. This represents a new, easy to construct, unsupervised ensemble learner, which we term Spectral Meta-Learner (SML).
An interest group of conspiring classifiers (a cartel), which maliciously attempts to veer the overall ensemble solution away from the (unknown) ground truth, leads to a rank-two covariance matrix. Furthermore, in contrast to majority voting, SML is robust to the presence of a small enough cartel whose members are unknown.
In addition, we demonstrate the advantages of spectral approaches based on these insights, using both simulated and real-world datasets. When the independence assumptions hold approximately, SML is typically better than most classifiers in the ensemble and their majority vote, achieving results comparable to the maximum likelihood estimator (MLE). Empirically, we find SML to be a better starting point for computing the MLE that consistently leads to improved performance. Finally, spectral approaches are also robust to cartels and therefore helpful in analyzing surveys where a biased sub-group of advisers (a cartel) may have corrupted the data.
For simplicity, we consider the case of questions with yes/no answers.
Hence, the advisers, or algorithms, provide to each query only one of two possible answers, either (positive) or (negative).
Following standard statistical terminology, the advisers or algorithms are called binary classifiers, and their answers are termed predicted class labels .
Each question is represented by a feature vector
. Each question is represented by a feature vectorcontained in a feature space .
In detail, let be binary classifiers of unknown reliability, each providing predicted class labels to a set of instances , whose vector of true (unknown) class labels is denoted by . We assume that each classifier was trained in a manner undisclosed to us using its own labeled training set, which is also unavailable to us. Thus, we view each classifier as a black-box function of unknown classification accuracy.
Using only the predictions of the binary classifiers on the unlabeled set and without access to any labeled data, we consider the two problems stated in the introduction: i) rank the performances of the classifiers; and ii) combine their predictions to provide an improved estimate of the true class label vector .
We represent an instance and class label pair as a random vector
with probability density function
as a random vector with probability density function, and with marginals and
In the present study, we measure the performance of a binary classifier by its balanced accuracy , defined as
where and are its sensitivity (fraction of correctly predicted positives) and specificity (fraction of correctly predicted negatives). Formally, these quantities are defined as
In our analysis we make the following two assumptions: i) The unlabeled instances are i.i.d. realizations from the marginal distribution ; and ii) the classifiers are conditionally independent, in the sense that prediction errors made by one classifier are independent of those made by any other classifier. Namely, for all , and for each of the two class labels, with
Classifiers that are nearly conditionally independent may arise, for example, from advisers who did not communicate with each other, or from algorithms that are based on different design principles or independent sources of information.
Note that these assumptions appear also in other works considering a setting similar to ours [15, 23] , as well as in supervised learning, in the development of classifiers (e.g., Naïve Bayes) and ensemble methods
, as well as in supervised learning, in the development of classifiers (e.g., Naïve Bayes) and ensemble methods.
Ranking of classifiers
To rank the classifiers without any labeled data, in this paper we present a spectral approach based on the covariance matrix of the classifiers. To motivate our approach it is instructive to first study its asymptotic structure as the number of unlabeled test data tends to infinity, . Let be the population covariance matrix of the classifiers, whose entries are defined as
where denotes expectation with respect to the density and .
The following lemma, proven in the supplementary information, characterizes the relation between the matrix and the balanced accuracies of the classifiers:
The entries of are equal to
where is the class imbalance,
The key insight from this lemma is that the off-diagonal entries
of are identical to those of a rank-one matrix
with unit-norm eigenvector and eigenvalue
Importantly, up to a sign ambiguity, the entries of are proportional to the balanced accuracies of the classifiers,
Hence, the classifiers can be ranked according to their balanced accuracies by sorting the entries of the eigenvector .
While typically neither nor are known, both can be estimated from the finite unlabeled dataset . We denote the corresponding sample covariance matrix by . Its entries are
In particular, and asymptotically as . Hence, for a sufficiently large unlabeled set , it should be possible to accurately estimate from the eigenvector and consequently the ranking of the classifiers.
One possible approach is to construct an estimate of the rank one matrix and then compute its leading eigenvector. Given that , for all we may estimate , and we only need to estimate the diagonal entries of . A computationally efficient way to do this, by solving a set of linear equations, is based on the following observation: upon the change of variables , we have for all ,
Hence, if we knew we could find the vector by solving the above system of equations. In practice, as we only have access to we thus look for a vector with small residual error in the above equations. We then estimate the diagonal entries by and proceed with eigendecomposition of . Further details on this and other approaches to estimate appear in the Supplementary Information.
Next, let us briefly discuss the error in this approach. First, since as , it follows that and consequently . Hence, asymptotically we perfectly recover the correct ranking of the classifiers. Since is rank-one, and both and are symmetric, as shown in the Supplementary Information, the leading eigenvector is stable to small perturbations. In particular, Finally, note that if all classifiers are better than random and the class imbalance is bounded away from , then we have a large spectral gap with .
The Spectral Meta Learner (SML)
Next, we turn to the problem of constructing a meta-learner expected to be more accurate than most (if not all) of the classifiers in the ensemble. In our setting, this is equivalent to estimating the unknown labels by combining the labels predicted by the classifiers.
The standard approach to this task is to determine for all the unlabeled instances the maximum likelihood estimator (MLE) of their true class labels . Under the assumption of independence between classifier errors and between instances, the overall likelihood is the product of the likelihoods of the individual instances, where the likelihood of a label for an instance is
As shown in the Supplementary Information, the MLE can be written as a weighted sum of the binary labels , with weights that depend on the sensitivities and specificities of the classifiers. For an instance ,
Eq. (11) shows that the MLE is a linear ensemble classifier, whose weights depend, unfortunately, on the unknown specificities and sensitivities of the classifiers.
The common approach, pioneered by Dawid and Skene , is to look for all labels and classifier specificities and sensitivities that jointly maximize the likelihood. Given an estimate of the true class labels, it is straightforward to estimate each classifier sensitivity and specificity.
Similarly, given estimates of and , the corresponding estimates of are easily found via (11 ). Hence,
the MLE is typically approximated by expectation-maximization (EM)
). Hence, the MLE is typically approximated by expectation-maximization (EM)[18, 19, 20, 21, 23].
As is well known, the EM procedure is guaranteed to increase the likelihood at each iteration till convergence. However, its key limitation is that due to the non-convexity of the likelihood function, the EM iterations often converge to a local (rather than global) maximum.
Importantly, the EM procedure requires an initial guess of the true labels . A common choice is the simple majority rule of all classifiers. As noted in previous studies, majority voting may be suboptimal, and starting from it, the EM procedure may converge to suboptimal local maxima . Thus, it is desirable, and sometimes crucial, to initialize the EM algorithm with an estimate that is close to the true label .
Using the eigenvector described in the previous section, we now present a novel construction of an initial guess that is typically more accurate than majority voting. To this end, note that a Taylor expansion of the unknown coefficients and in (12) around gives, up to second order terms ,
Recall that by Lemma 1, up to a sign ambiguity the entries of the leading eigenvector of are proportional to the balanced accuracies of the classifiers, . This sign ambiguity can be easily resolved if we assume, for example, that most classifiers are better than random. Replacing in (14) by the eigenvector entries of an estimate of yields a novel spectral-based ensemble classifier, which we term the Spectral Meta-Learner (SML),
Intuitively, we expect SML to be more accurate than majority voting as it attempts to give more weight to more accurate classifiers. Lemma 4 in the Supplementary Information provides insights on the improved performance achieved by SML in the special case when all algorithms but one have the same sensitivity and specificity. Numerical results for more general cases are described in the simulation section, where we also show that empirically, on several real data problems, SML provides a better initial guess than majority voting for EM procedures that iteratively estimate the MLE.
Learning in the Presence of a Malicious Cartel
Consider a scenario whereby a small fraction of the classifiers belong to a conspiring cartel (e.g., representing a junta or an interest group), maliciously designed to veer the ensemble solution toward the cartel’s target and away from the truth. The possibility of such a scenario raises the following question: how sensitive are SML and majority voting to the presence of a cartel? In other words, to what extent can these methods ignore, or at least substantially reduce, the effect of the cartel classifiers without knowing their identity?
To this end, let us first introduce some notation. Let the classifiers be composed of a subset of “honest” classifiers and a subset of malicious cartel classifiers. The honest classifiers satisfy the assumptions of the previous section: each classifier attempts to correctly predict the truth with a balanced accuracy , and different classifiers make independent errors. The cartel classifiers, in contrast, attempt to predict a different target labeling, . We assume that conditional on both the cartel’s target and the true label, the classifiers in the cartel make independent errors. Namely, for all , and for any labels
Similarly to the previous sections, we assume that the prediction errors of cartel and honest classifiers are also (conditionally) independent.
The following lemma, proven in the supplementary information, expresses the entries of the population covariance matrix in terms of the following quantities: the balanced accuracies of the classifiers, the balanced accuracy of the cartel’s target with respect to the truth, and the balanced accuracies of the cartel members relative to their target.
Given honest classifiers and classifiers of a cartel , the entries of satisfy
where is the class imbalance, as in (6).
Next, the following theorem shows that in the presence of a single cartel, the off-diagonal entries of correspond to a rank-two matrix. We conjecture that in the presence of independent cartels, the respective rank is .
Given honest classifiers and classifiers belonging to a cartel, , the off-diagonal entries of correspond to a rank-two matrix with eigenvalues
and, with , ,
As an illustrative example of Theorem 1, consider the case where the cartel’s target is unrelated to the truth, i.e. . In this case , so , and
Next, according to (15) SML weighs each classifier by the corresponding entry in the leading eigenvector. Hence, if the cartel’s target is orthogonal to the truth ) and , SML asymptotically ignores the cartel (Fig. S3). In contrast, regardless of , majority voting is affected by the cartel, proportionally to its fraction size . Hence, SML is more robust than majority voting to the presence of such a cartel.
Application to simulated and real-world datasets
The examples provided in this section showcase strengths and limitations of spectral approaches to the problem of ranking and combining multiple predictors without access to labeled data. First, using simulated data of an ensemble of independent classifiers and an ensemble of independent classifiers containing one cartel, we confirm the expected high performance of our ranking and SML algorithms. In the second part we consider the predictions of 33 machine learning algorithms as our ensemble of binary classifiers, and test our spectral approaches on 17 real-world datasets collected from a broad range of application domains.
We simulated an ensemble of independent classifiers providing predictions for instances, whose ground truth had class imbalance . To imitate a difficult setting, where some classifiers are worse than random, each generated classifier had different sensitivity and specificity chosen at random such that its balanced accuracy was
uniformly distributed in the interval
. To imitate a difficult setting, where some classifiers are worse than random, each generated classifier had different sensitivity and specificity chosen at random such that its balanced accuracy was uniformly distributed in the interval. We note that classifiers that are worse than random may occur in real studies, when the training data is too small in size or not sufficiently representative of the test data. Finally, we considered the effect of a malicious cartel consisting of of the classifiers, having their own target labeling. More details about the simulations are provided in the Supplementary Information.
Ranking of Classifiers: We constructed the sample covariance matrix, corrected its diagonal as described in the Supplementary Information and computed its leading eigenvector . In both cases (independent classifiers and cartel), with probability of at least , the classifier with highest accuracy was also the one with the largest entry (in absolute value) in the eigenvector , and with probability its inferred rank was among the top five classifiers (Fig. S4). Note that even if the test data of size were fully labeled, identifying the best performing classifier would still be prone to errors, as the estimated balanced accuracy has itself an error of
Unsupervised Ensemble-Learning: Next, for the same set of simulations we compared the balanced accuracy of majority voting and of SML. We also considered the predictions of these two meta-learners as starting points for iterative EM calculation of the MLE (iMLE). As shown in Fig. 1, SML was significantly more accurate than majority voting. Furthermore, applying an EM procedure with SML as an initial guess provided relatively small improvements in the balanced accuracy. Majority voting, in contrast, was less robust. Moreover, in the presence of a cartel, computing the MLE with majority voting as its starting point exhibited a multi-modal behavior, sometimes converging to a local maxima with a relatively low balanced accuracy.
A more detailed study of the sensitivity of SML and majority voting and their respective iMLE solutions versus the size of a malicious cartel with is shown in Fig. S5. As expected, the average balanced accuracy of all methods decreases as a function of the cartel’s fraction , and once the cartel’s fraction is too large all approaches fail. In our simulations, both SML and iMLE initialized with SML were far more robust to the size of the cartel than either majority voting or iMLE initialized with majority voting. With a cartel size of 20%, SML was still able to construct a nearly perfect predictor, whereas the balanced accuracy of majority voting and iMLE initialized with majority voting were both far from 1. Interestingly, in our simulations, iMLE using SML as starting condition showed no significant improvement relative to the average balanced accuracy of SML itself.
We applied our spectral approaches to 17 different datasets of moderate and large sizes from medical, biological, engineering, financial and sociological applications. Our ensemble of predictions was comprised of 33 machine-learning methods available in the software package Weka  (see Methods). We split each dataset into a labeled part and an unlabeled part, the latter serving as the test data used to evaluate our methods. To mirror our problem setting, each algorithm had access and was trained on different subsets of the labeled data (see Supplementary Information).
Figs. 2, S6, S7 and S8 show the results of different meta-classifiers on these datasets. Let us now interpret these results and explain the apparent differences in balanced accuracy between different approaches, in light of our theoretical analysis in the previous section.
In datasets where our assumptions are approximately satisfied, we expect SML, iMLE initialized with SML, and iMLE initialized with majority voting to exhibit similar performances. This is the case in the ACS data (left panel of Fig. 2), and in all datasets in Fig. S6. We verified that in these datasets (3) indeed holds approximately (see Table S3). In addition, in all these datasets, the corresponding sample covariance matrix of the 33 classifiers was almost rank-one with .
Fig. S7 and Fig. 2 (central panel) correspond to datasets where the median performance of the classifiers was only slightly above 0.5, with some classifiers having poor, even worst than random balanced accuracy. Interestingly, in these datasets, the covariance matrix between classifiers was far from being rank one (similar to the case when cartels were present). The relative amount of variance captured by the first two leading eigenvalues and was, on average, and , respectively. In these datasets, SML seems to offer a clear advantage: initializing iMLE with SML rather than with majority voting avoids the poor outcomes observed in the NYSE, AMEX and PNS datasets.
Finally, the datasets in Fig. S8 and in Fig. 2 (right panel) are characterized by very sparse (ENRON) or high-dimensional (LASTFM) feature spaces . In these datasets, some instances were highly clustered in feature space, while others were isolated. Thus, in these datasets many classifiers made identical errors.
Remarkably, even in these cases, iMLE initialized with SML had an equal or higher median balanced accuracy than iMLE initialized with majority voting. This was consistent across all datasets, indicating that the SML prediction provided a better starting point for iMLE than majority voting.
Summary and Discussion
In this paper we presented a novel spectral-based statistical analysis for the problems of unsupervised ranking and combining of multiple predictors. Our analysis revealed that under standard independence assumptions, the off-diagonal of the classifiers covariance matrix corresponds to a rank-one matrix, whose eigenvector entries are proportional to the classifiers balanced accuracies. To the best of our knowledge, our work gives the first computationally efficient and asymptotically consistent solution to the classical problem posed by Dawid and Skene  in 1979, for which thus far only non-convex iterative likelihood maximization solutions have been proposed [18, 26, 27, 28, 29].
Our work not only provides a principled spectral approach for unsupervised ensemble learning (such as our SML), but also raises several interesting questions for future research.
First, our proposed spectral-based SML has inherent limitations: it may be sub-optimal for finite samples, in particular when one classifier is significantly better than all others. Furthermore, most of our analysis was asymptotic in the limit of an infinitely large unlabeled test set, and assuming perfect conditional independence between classifier errors. A theoretical study of the effects of a finite test set, and of approximate independence between classifiers on the accuracy of the leading eigenvector is of interest. This is particularly relevant in the crowdsourcing setting, where only few entries in the prediction matrix are observed. While an estimated covariance matrix can be computed using the joint observations for each pair of classifiers, other approaches that directly fit a low rank matrix may be more suitable.
Second, a natural extension of the present work is to multi-class or regression problems where the response is categorical or continuous, instead of binary. We expect that in these settings the covariance matrix of independent classifiers or regressors is still approximately low-rank. Methods similar to ours may improve the quality of existing algorithms.
Third, the quality of predictions may also be improved by taking into consideration instance difficulty, discussed for example in [18, 23]. These studies assume that some instances are harder to classify correctly, independent of the classifier employed, and propose different models for this instance difficulty. In our context, both very easy examples (on which all classifiers agree) and very difficult ones (on which classifier predictions are as a good as random) are not useful for ranking the different classifiers. Hence, modifying our approach to incorporate instance difficulty is a topic for future research.
Finally, our work also provides insights on the effects of a malicious cartel. The study of spectral approaches to identify cartels and their target, as well as to ignore their contributions, is of interest due to its many potential applications, such as electoral committees and decision-making in trading.
Materials and methods
Datasets and Classifiers
Statistical Analysis and Visualization
Statistical analysis and visualization were performed using MATLAB (2012a, The MathWorks, Natick, MA) and R (www.R-project.org). Additional information is provided in the Supplementary Information.
We thank Amit Singer, Alex Kovner, Ronald Coifman, Ronen Basri and Joseph Chang for their invaluable feedback. The Wisconsin breast cancer dataset was collected at the University of Wisconsin Hospitals by Dr. W.H. Wolberg and colleagues. F.S. is supported by the American-Italian Cancer Foundation. B.N. is supported by grants from the Israeli Science Foundation (ISF) and from Citi Foundation. Y.K is supported by the Peter T. Rowley Breast Cancer Research Projects (New York State Department of Health) and NIH (R0-1 CA158167).
Tsai, C.-F & Hsiao, Y.-C.
(2010) Combining multiple feature selection methods for stock prediction: Union, intersection, and multi-intersection approachesDecis. Support Syst. 50, 258-269.
-  Zweig, J. (2011) Making sense of market forecasts, Wall Street Journal - Jan 8, 2011.
European Central Bank.
Survey of professional forecasters.
-  Federal Reserve Bank of Philadelphia. Survey of professional forecasters. www.philadelphiafed.org/research-and-data/real-time-center/survey-of-professional-forecasters/
-  Monetary Authority of Singapore. Survey of professional forecasters. www.mas.gov.sg/news-and-publications/surveys/mas-survey-of-professional-forecasters.aspx
-  Genre, V, Kenny, G, Meyler, A, & Timmermann, A. G. (2010) Combining the forecasts in the ECB survey of professional forecasters: Can anything beat the simple average?, (Social Science Research Network, Rochester, NY), SSRN Scholarly Paper ID 1719622.
-  Micsinai, M, et al. (2012) Picking ChIP-seq peak detectors for analyzing chromatin modification experiments Nucleic acids research 40, e70. PMID: 22307239.
-  Wright, F, De Vito, C, Langer, B, & Hunter, A. (2007) Multidisciplinary cancer conferences: A systematic review and development of practice standards European Journal of Cancer 43, 1002–1010.
-  Cole, S, et al. (1978) Peer review in the national science foundation: Phase one of a study., (Office of Publications, National Academy of Sciences, 2101 Constitution Ave., N.W., Washington, D.C. 20418 (ISBN 0-309-02788-8))
-  Margolin, A. A, et al. (2013) Systematic Analysis of Challenge-Driven Improvements in Molecular Prognostic Models for Breast Cancer Science Translational Medicine 5, 181
-  Linstone, H & Turoff, M. e. (1975) Delphi Method : Techniques and Applications. (Addison-Wesley Pub. Co)
-  Stock, J. H & Watson, M. W. (2006) Forecasting with many predictors, (Elsevier), Handbook of economic forecasting.
-  Timmermann, A. (2006) in Handbook of Economic Forecasting, eds. G. Elliott, C. G & Timmermann, A. (Elsevier) Volume 1, pp. 135–196.
-  Degroot, MH, (1974) Reaching a concensus, J. Amer. Stat. Assoc., 69, 118-121.
-  Dawid, A. P & Skene, A. M. (1979) Journal of the Royal Statistical Society. Series C (Applied Statistics) 28, 20–28.
-  Karger, D. R, Oh, S, & Shah, D. (2011) Budget-optimal crowdsourcing using low-rank matrix approximations. Proc. of the IEEE Allerton Conf. on Communication, Control, and Computing, pp. 284–291.
-  Karger, D. R, Oh, S, & Shah, D. (2011) Iterative Learning for Reliable Crowdsourcing Systems NIPS 1953-1961.
-  Whitehill, J, Ruvolo, P, Wu, T, Bergsma, J, & Movellan, J. (2009) Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. NIPS.
-  Padhraic Smyth, U. F. (1996) Inferring Ground Truth from Subjective Labelling of Venus Images
-  Sheng, V. S, Provost, F, & Ipeirotis, P. G. (2008) Get another label? improving data quality and data mining using multiple, noisy labelers, KDD ’08. (ACM, New York, NY, USA), p. 614–622.
-  Welinder, P, Branson, S, Belongie, S, & Perona, P. (2010) in Advances in Neural Information Processing Systems 23, eds. Lafferty, J, Williams, C, Shawe-Taylor, J, Zemel, R, & Culotta, A. pp. 2424–2432.
-  Yan, Y, et al. (2010) Modeling annotator expertise: Learning when everybody knows a bit of something. (AISTATS 2012) pages 932–939, 2010
-  Raykar, V. C, et al. (2010) Learning From Crowds J. Mach. Learn. Res. 11, 1297–1322.
-  Dietterich, T. G. (2000) Ensemble Methods in Machine Learning. (Springer), p. 1–15.
-  Witten, I. H, Frank, E, & Hall, M. A. (2011) Data mining: practical machine learning tools and techniques. (Morgan Kaufmann, Burlington, MA).
-  Jin, R & Ghahramani, Z. (2003) Learning with Multiple Labels. in: S. Becker, S. Thrun, K. Obermayer (Eds.), Advances in Neural Information Processing Systems, vol. 15, MIT Press, Cambridge, MA, 2003, pp. 897–904.
-  Lauritzen, S. L. (1995) ”The EM algorithm for graphical association models with missing data.” Computational Statistics & Data Analysis 19.2: 191-201.
-  Snow, R, O’Connor, B, Jurafsky, D, & Ng, A. Y. (2008) Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks, EMNLP ’08. (Association for Computational Linguistics, Stroudsburg, PA, USA), p. 254–263.
-  Walter, S. D & Irwig, L. M. (1988) Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review Journal of clinical epidemiology 41, 923–937. PMID: 3054000.
-  Strino, F, Parisi, F, & Kluger, Y. (2011) VDA, a method of choosing a better algorithm with fewer validations PloS one 6, e26074. PMID: 22046256.
A Covariance between classifiers
Proof of Lemma 1. To prove the lemma we first compute the mean and variance of the -th classifier. We then use these results to compute the entries of the population covariance matrix, .
Under the assumption of independence between instances, the population mean of the -th classifier is
Using the definitions of sensitivity , specificity , and class imbalance , the equation above can be expressed as follows,
where is the balanced accuracy of the -th classifier and .
Similarly, the population variance of the -th classifier is
Next, consider . Under the assumption of independence of errors between different instances and between different classifiers, for
B Rank-one Eigenvector Estimation
In this section we describe four approaches to estimate the eigenvector of the rank one matrix from the sample covariance matrix . We term these methods (i) linear system approach; (ii) weighted linear system approach; (iii) SDP approach; and (iv) direct eigendecomposition approach. In our simulations we found that all four approaches gave comparable rankings, though the latter was slightly less accurate (Fig. S1). The linear system approach (i) had computational complexity comparable to the fastest method of direct eigendecomposition, while providing a ranking of quality comparable to the much more computationally heavy SDP method. Method (i) was also slightly faster to compute than its weighted counterpart, method (ii), so we chose it for our benchmarks.
b.1 Linear system
As discussed in the main text, one approach to rank the classifiers is to construct an estimator of the rank-one matrix , compute its leading eigenvector and rank the classifiers by sorting its entries. Given that , we estimate the off-diagonal entries of simply by those of , and only need a consistent method to estimate the diagonal entries. To this end, note that upon the change of variables , it follows that in the population setting, for all ,
In the finite sample setting, we replace the unknown by and look for an -dimensional vector such that the relation above holds approximately for all pairs ,
From the vector we estimate the diagonal entries of as .
As the functional in Eq. (S5) is quadratic, the vector is efficiently found by solving a system of linear equations with unknowns. Since as sample size , it follows that is an asymptotically consistent estimate of . Consequently the resulting is a consistent estimate of , and asymptotically it yields a perfectly correct ranking of the classifiers, according to their balanced accuracies.
b.2 Weighted linear system
Similar to the linear system approach presented above, we can instead consider the following weighted least square problem, where is given by Eq. (9) from the main text.
The resulting estimator is also solved via a system of linear equations.
b.3 SDP approach
Here we look for a rank-one matrix , whose off-diagonal terms are closest to those of . While the rank-one constraint is non-convex, its standard relaxation to a trace constraint yields
subject to , and where is a suitably chosen regularization parameter. This is a convex problem, which can be solved via semi-definite programming . We thus term it SDP approach. While in principle SDP problems can be solved to arbitrary accuracy in polynomial time in , this approach is significantly slower than the two previous ones, which require solutions to systems of linear equations.
b.4 Direct eigendecomposition
Finally, an even simpler approach is to rank the classifiers by directly computing the leading eigenvector of . For a finite number of classifiers , it follows from Lemma 1 that as , this direct eigen-decomposition approach is generally not consistent. However, as the following lemma shows, if the rank one matrix has a large spectral gap, , then this leading eigenvector is close to the true one.
Let be the leading unit-norm eigenvector of the population matrix , and let be given by Eq. (7) in the main text. Then,
Proof : Let be the leading eigenvalue of with corresponding unit-norm eigenvector . Let be the eigenvalue of the rank-one matrix with corresponding unit-norm eigenvector . First, note that
where is a diagonal matrix with entries
Hence . It thus readily follows from Weyl’s theorem that
Now, multiplying the eigenvector equation from the left by , and inserting the relation (S9) gives that
The lemma follows by combining Eq. (S10) with the bound . .
Note that if all classifiers in the ensemble have a balanced accuracy bounded away from , then and then for the angle between and is small.
Finally, we note that this direct eigendecomposition approach is equivalent to ranking classifiers by a singular value decomposition (SVD) of the
Finally, we note that this direct eigendecomposition approach is equivalent to ranking classifiers by a singular value decomposition (SVD) of themean-centered matrix of predicted labels . This approach, although apparently without the mean-centering operation, was recently suggested in , which proposed the -th entry in the leading right singular vector as a proxy for the reliability of the -th classifier. Our work provides a novel probabilistic interpretation to this approach, as it shows that the entries of , which is also the leading right singular vector of the (mean-centered) matrix , are approximately those of , which in turn are proportional to the balanced accuracies of the classifiers.
b.5 Asymptotic Eigenvector Stability
We now consider the asymptotic stability of the estimated eigenvector to small perturbations due to finite sample fluctuations in our estimate . First note that for all , . It thus follows that upon solving the linear system for the vector , asymptotically its errors are also , and hence for all , we may assume that .
To understand how these fluctuations affect the estimation of the leading eigenvector of the rank one matrix , we consider the one-parameter family of matrices where is a matrix whose entries are all . By definition, at we have that . We thus view as a small parameter, study the dependence of the leading eigenvector of on , and eventually plug in .
Given that both and are symmetric, standard results from matrix perturbation theory  imply that for sufficiently small the leading eigenvector and eigenvalue of are analytic functions of . At , these resort to the eigenvector and eigenvalue of the exact rank one matrix . For small we may thus expand
Inserting this expansion into the eigenvalue-eigenvector equation , and equating powers of gives that the equation reads
Since the eigenvector is defined only up to a normalization constant, we conveniently chose it to be that for all , which in particular implies that .
Now multiplying Eq. (S11) from the left by gives that and
where denotes the Moore-Penrose pseudo-inverse of .
The key point from Eq. (S12) is that for a given spectral gap of size of the rank-one matrix , asymptotically in , the perturbation in the leading eigenvector estimate is .
C Spectral Meta-Learner
In this section we present the derivation of the Spectral Meta-Learner (SML) as a linearization of the maximum likelihood estimator (MLE) of the vector of true class labels around .
c.1 Maximum Likelihood Estimator (MLE)
Under the assumption of independence between classifier errors and between instances, given the specificities and sensitivities of the classifiers, the overall likelihood of the labels of all instances is a product of the likelihood of each individual instance label. Hence, for each instance its class label can be estimated independently of the class labels of all other instances. The MLE of is
Next, note that the conditions and in the two sums above can be represented by the following two indicator functions,
Using these indicator functions allows to express the MLE as a function of and as follows