Diabetic retinopathy (DR) is a serious eye disease that originates from diabetes mellitus and is the most common cause of blindness in the developed countries. Early treatment can prevent patients to become affected from this condition or at least the progression of DR can be slowed down. Thus, mass screening of patients suffering from diabetes is highly desired, but manual grading is slow and resource demanding. Therefore, several efforts have been made to establish reliable computer-aided screening systems based on color fundus images . The promising results reported by Fleming et al.  and Jelinek et al.  indicates that automatic DR screening systems are getting closer to be used in clinical settings.
A key feature to recognize DR is to detect microaneurysms (MAs) in the fundus of the eye. The importance of handling MAs are two-fold. First, they are normally the earliest sign of DR, hence their timely and precise detection is essential. On the other hand, the grading performance of computer-aided DR screening systems highly depends on MA detection  . In this paper, we propose a microaneurysm detector which provides remarkable results from both aspects.
One way to ensure high reliability and raise accuracy in a detector is to consider ensemble-based systems, which have been proven to be efficient in several fields. However, the usual ensemble techniques aim to combine class labels or real values which cannot be adopted in our case. In MA detection, detectors provide spatial coordinates as centers of potential MA candidates. The use of well-known ensemble techniques would require a classification of each pixel, which can be misleading in our context, since different algorithms extract MAs with different approaches and the MA centers may not coincide exactly. To overcome this difficulty, we gather close MA candidates of the individual detectors and apply a voting scheme on them.
In , Niemeijer et al. showed that the fusion of the results of the several MA detectors lead to an increased average sensitivity measured at seven predefined false positive rates. In this paper, we propose a framework to build MA detector ensembles based on the combination of the internal components of the detectors not only on their output as in . Some of our earlier research on combining MA detectors did not provide reassuring results . To increase the accuracy of such ensembles, we must identify the weak points of MA detection. The first difficulty originates from the shape characteristics of MAs. They appear as small circular dark spots on the surface of the retina (see Figure 1), which can be hard to distinguish from fragments of the vascular system or from certain eye features. Most MA detectors tackle this problem in the following way: first, the green channel of the fundus image is extracted and preprocessed to enhance MA like characteristics. Then, in a coarse level step (which will be referred as candidate extraction in the rest of the paper), all MA-like objects are detected in the image. Finally, a fine level algorithm (usually a supervised classifier) removes the potentially false detections based on some assumptions about MAs. Our former investigations showed that the low sensitivity of MA detectors originates from the candidate extractor part . However, we could increase the sensitivity by applying proper preprocessing methods before candidate extraction. This technique causes a slight increment in the number of false positives, but it can be decreased by classification or voting.
In this paper, we propose an effective microaneurysm detector based on the combination of preprocessing methods and candidate extractors. We provide an ensemble creation framework to select the best combination. An exhaustive quantitative analysis is also given to prove the superiority of our approach over individual algorithms. We also investigate the grading performance of our method, which is proven to be competitive with other screening systems.
The rest of the paper is organized as follows: the selected preprocessing methods and candidate extractors are presented in section II and III, respectively. The details of the proposed ensemble creation framework is discussed in section IV. We present our evaluation methodology in section V. In section VI, we summarize our experimental results. A detailed discussion is given in section VII to address several issues. Finally, we draw conclusions in section VIII.
Ii Preprocessing methods
In this section, we present the selected preprocessing methods, which we consider to be applied before executing MA candidate extraction. The selection of the preprocessing method and candidate extractor components for this framework is a challenging task. Comparison of preprocessing methods dedicated to microaneurysm detection has not been published yet. Since preprocessing methods need to be highly interchangeable, we must select algorithms which can be used before any candidate extractor and do not change the characteristics of the original images (unlike e.g. shade correction ). We also found some techniques to generate too noisy images for MA detection (histogram equalization , adaptive histogram equalization  or color normalization ). Thus, we have selected methods which are well-known in medical image processing and preserve image characteristics. Naturally, the proposed system can be improved in the future with adding new methods. A summary on the key differences of the algorithms is given in Table I.
Ii-a Walter-Klein contrast enhancement 
This preprocessing method aims to enhance the contrast of fundus images by applying a gray level transformation using the following operator:
where are the intensity levels of the original and the enhanced image, respectively, is the mean value of the original grayscale image and is a transition parameter.
Ii-B Contrast limited adaptive histogram equalization 
Contrast limited adaptive histogram equalization (CLAHE) is a popular technique in biomedical image processing, since it is very effective in making the usually interesting salient parts more visible. The image is split into disjoint regions, and in each region a local histogram equalization is applied. Then, the boundaries between the regions are eliminated with a bilinear interpolation.
Ii-C Vessel removal and extrapolation 
We investigate the effect of processing images with the complete vessel system being removed based on the idea proposed in . We extrapolate the missing parts to fill in the holes caused by the removal using the inpainting algorithm presented in . MAs appearing near vessels become more easily detectable in this way.
Ii-D Illumination equalization 
This preprocessing method aims to reduce the vignetting effect caused by uneven illumination of retinal images. Each pixel intensity is set according to the following formula:
where are the original and the new pixel intensity values, respectively, is the desired average intensity and is the local average intensity. MAs appearing on the border of the retina are enhanced by this step.
Ii-E No preprocessing
We also consider the results of the candidate extractors obtained for the original images without any preprocessing. That is, we formally consider a ”No preprocessing” operation, as well.
|Walter-Klein||contrast enhancement||gray level transformation|
|CLAHE||salient object enhancement||local histogram equalization|
|Vessel Removal||MA enhancement near vessels||vessel removal and inpainting|
|Illumination eq.||MA enhancement at the border of the ROI||vignette correction|
Iii Microaneurysm candidate extractors
Candidate extraction is a process which aims to spot any objects in the image showing MA-like characteristics. Individual MA detectors consider different principles to extract MA candidates. In this section, we provide a brief overview of the candidate extractors involved in our analysis. Again, just as for preprocessing methods, adding new MA candidate extractors may lead to further improvement in the future. A summary on the key differences of the candidate extractor algorithms and their performance measured in the ROC training dataset  are shown in Table II.
Iii-a Walter et al. 
Candidate extraction is accomplished by grayscale diameter closing. That is, this method aims to find all sufficiently small dark patterns on the green channel. Finally, a double threshold is applied.
Iii-B Spencer et al. 
From the input fundus image, the vascular map is extracted by applying twelve morphological top-hat transformations with twelve rotated linear structuring elements (with a radial resolution
). Then, the vascular map is subtracted from the input image, which is followed by the application of a Gaussian matched filter. The resulting image is then binarized with a fixed threshold. Since the extracted candidates are not precise representations of the actual lesions, a region growing step is also applied to them. While the original paper is written to detect MAs on fluorescein angiographic images, our implementation is based on the modified version published by Fleming et al. .
Iii-C Circular Hough-transformation 
Following the idea presented in , we established an approach based on the detection of small circular spots in the image. Candidates are obtained by detecting circles on the images using circular Hough transformation. With this technique, a set of circular objects can be extracted from the image.
Iii-D Zhang et al. 
In order to extract candidates, this method constructs a maximal correlation response image for the input retinal image. This is accomplished by considering the maximal correlation coefficient with five Gaussian masks with different standard deviations for each pixel. The maximal correlation response image is thresholded with a fixed threshold value to obtain the candidates. Vessel detection and region growing is applied to reduce the number of candidates, and to determine their precise size, respectively.
Iii-E Lazar et al. 
Pixel-wise cross-section profiles with multiple orientations are used to construct a multi-directional height map. This map assigns a set of height values that describe the distinction of the pixel from its surroundings in a particular direction. In a modified multilevel attribute opening step, a score map is constructed from which the MAs are extracted by thresholding.
|Algorithm||Method||Sensitivity||FP / I|
|Zhang||matching multiple Gaussian masks||33%||328.3|
|Lazar||cross-section profile analysis||48%||73.94|
Iv Ensemble creation
In this section, we describe our ensemble creation approach. In our framework, an ensemble is a set of preprocessing method, candidate extractor or shortly pairs. The meaning of a preprocessing method, candidate extractor pair is that first we apply the preprocessing method to the input image and then we apply the candidate extractor to this result. That is, such a pair will extract a set of candidates from the original image. If an ensemble contain more preprocessing method, candidate extractor pairs, their outputs are fused in the following way: for each candidate , all such candidates of the other participants are collected, whose euclidean distance is smaller than a predefined constant from . Let denote that the set of these points collected for a candidate . Then, the centroid calculated from is put into .
Ensemble creation is a process where all ensembles from an ensemble pool is evaluated and the best performing one regarding an evaluation function on a training set is selected. To evaluate an ensemble , its output candidate set must be compared to the ground truth in the following way: if for a exists a point in the ground truth, whose euclidean distance from is smaller than a predefined constant , then is considered as a true positive. Otherwise, is false positive, while each ground truth point is a false negative which does not have a close candidate from .
The selection of the optimal ensemble would require each possible preprocessing method, candidate extractor ensembles to be evaluated to find the optimal one. However, currently we consider preprocessing methods and candidate extractors in our experiments. That is, we have 25 preprocessing method, candidate extractor pairs with number of possible combinations to form the ensemble. It would be very resource-demanding to evaluate such a large number of combinations, so we used simulated annealing  as a search algorithm to find the final ensemble, which is proven to be effective in such large search spaces. However, we describe the selection procedure as an exhaustive search in the latter parts, since it is better to evaluate all configurations if enough resources are available, and several other choices of search algorithms are possible.
As an energy function, we used the competition performance metric CPM , which is defined as the average sensitivity level at seven predefined false positive per image rate () . The process of ensemble creation is also shown in Figure 2.
The ensemble creation part results in a set of preprocessing method, candidate extractor pairs. This ensemble then can be used to detect MAs on unknown images. The final ensemble is applied in real detection in the same way as in the training phase. Namely, the final MAs are detected by the fusion of the MA candidates of the individual pairs building up the ensemble . Similarly, for every detected MA we will have a confidence value as described above. Thus, for the final decision on the presence of MAs, the output MA set needs to be thresholded according to the assigned confidence values. The choice of the threshold value is discussed in section VII in detail.
The proposed ensemble creation method can be summarized through the following steps:
Algorithm 1: Selection of the optimal combination of preprocessing methods and candidate extractors.
We have evaluated the proposed approach for both MA detection and DR grading. In this section, we present the evaluation methodology we used in each case.
V-a MA detection
We have evaluated the MA detection capabilities of the proposed method in the ROC competition for MA detectors , as well as on a publicly available  and a private database. In this section, we provide a brief overview on these databases and on the methodology we used for the evaluation of MA detection performance of the proposed approach.
V-A1 Retinopathy Online Challenge (ROC) 
ROC is a worldwide competition dedicated to measure the accuracy of microaneurysm detectors. The ROC database consists of 50 training and 50 test images with different resolutions (, and ), FOV and JPEG compression. The average number of MAs for the training and test sets are 6.72 and 6.86, respectively. There are 13 and 10 images of the training and test sets, where no MAs are marked by the experts.
V-A2 DiaretDB1 2.1 database 
The DiaretDB1 2.1 database contains 28 losslessly compressed training and 61 test images with a resolution and FOV. The average number of MAs for the training and test sets are 4.34 and 3.91, respectively. There are 15 and 39 images of the training and test sets, where no MAs are marked by the experts.
V-A3 Private database provided by Moorfields Eye Hospital, UK
This database consists of 60 losslessly compressed images with a resolution and FOV. The average number of MAs for the training and test sets are 8.67 and 8.87, respectively. There are 10 and 8 images of the training and test sets, respectively, where no MAs are marked by the experts.
For each database, we provide the Free-response Receiver Operating Characteristic (FROC) curves , which plots the sensitivity against the average number of false positives per image. To measure the sensitivity at different average false positive per image levels, we thresholded the output set of the MA detector based on the confidence values assigned to each candidate. For the ROC dataset, we also provide the current ranking of the competition along with the CPM values (see section VIII for details) that serves as the basis for the ranking. In addition, we also calculated a partial AUC of the algorithms in the same range (between and ) by normalizing the average false positive per image figure by dividing with the maximum () and applying trapezoidal integration. The empirical AUC calculated this way is likely to underestimate the true AUC. However, the uncertainty for the partial AUCs may be quite high due to the low number of images.
V-B DR grading
We have also evaluated our ensemble-based approach to see its grading performance to recognize DR. For this aim, we determined the image-level classification rate of the ensemble on the Messidor111Kindly provided by the Messidor program partners (see http://messidor.crihan.fr). dataset containing 1200 images. That is, the presence of any MA means that the image contains signs of DR, while the absence of MAs indicates a healthy case. In other words, a pure yes/no decision of the system has been tested.
V-B1 Ensemble creation
As there is no training set provided for the Messidor database, we used an independent dataset (the ROC dataset) to train our algorithm. Note that, this is quite a strong handicap in comparison with the usual approach to train on a part of the same database. However, we feel that in this way we can get much closer to measure up the true performance of our system under real circumstances.
We used the publicly available Messidor database for testing. This database consists of 1200 losslessly compressed images with FOV and different resolutions (, and ). For each image, a grading score ranging from R0 to R3 is provided. These grades correspond to the following clinical conditions: a patient with an R0 grade has no DR. R1 and R2 are mild and severe cases of non-proliferative retinopathy, respectively. Finally, R3 is the most serious condition (proliferative retinopathy). The grading is based on the appearance of MAs, haemorrhages and neovascularization. The proportion of the images in the Messidor dataset: 540 R0 (46%), 153 R1 (12.75%), 247 R2 (20.58%) and 260 R3 (21.67%).
In our evaluation, we classified the retinal images whether they contain signs of DR (R1, R2, R3) or not (R0). The MA detector classifies an image as diseased if at least one MA was detected, and healthy otherwise. We measured the sensitivity, specificity and accuracy of the detector at different levels by thresholding the confidence values assigned to the MA candidates as described in section IV using the following formulas:
We also measured that the percentage of correctly recognized cases for each grade. We provided a fitted Receiver Operating Characteristic (ROC) curve along with the empirical and fitted AUC for the proposed method on the Messidor database. For curve fitting, we used JROCFIT .
In this section, we present our experimental results for both MA detection and DR grading.
Vi-a MA detection
In Table III, we exhibit the preprocessing method, candidate extractor pairs included in the selected ensembles for the three datasets, respectively. The rows of the table show the preprocessing methods from section II, while the columns label the candidate extractor algorithms listed in section III.
|Vessel Removal||D||R, D, M||R, D|
|Illumination eq.||R, M|
|No preprocessing||R||M||R, D||R|
Table IV contains the ranked quantitative results of the participants at the ROC competition, with the proposed ensemble (DRSCREEN) highlighted as the current leader. The performance of the ensemble is also shown in Figure 3 in terms of a FROC curve. As we can see from Table IV, the proposed ensemble earned both a higher CPM score and a higher partial AUC than the individual algorithms.
|Niemeijer et al.||0.395||0.469|
|Lazar et al.||0.355||0.449|
The FROC curves of the ensemble for the DiaretDB1 v2.1 and for the Moorfields database is shown in Figures 4 and 5, respectively. To the best of our knowledge, no corresponding quantitative results have been published for these databases yet. Thus, we disclose the results of the ensemble-based method only.
Vi-B DR grading
In Table V, we provide the sensitivity, specificity and accuracy measures of our detector corresponding to different threshold values, respectively. The fitted ROC curve of the detector can be seen in Figure 6. The empirical area under curve (AUC) is 0.875, while the AUC for the fitted curve is . Table V also contains the percentage of the correctly recognized cases for each class.
A strong point of the proposed method is that it performs well under difficult circumstances. Figure 7
shows an example image where the application of CLAHE made it easier to distinguish the MAs from their background. However, the use of the vessel removal and inpainting preprocessing method caused the missing of a true MA, while the detection of the remaining MA is easier in the absence of thin retinal vessels. Thus, using different preprocessing methods with candidate extractors creates diversity among the members of the ensemble, which is desired for systems using multiple estimators. This diversity ensures the suppression of false detections, since diverse detectors tend to make different mistakes. Thus, the false detections are likely to receive lower confidence values in the voting procedure.
Our experimental results show that the proposed ensemble-based MA detector outperforms the current individual approaches in MA detection. It has been also proven that the framework has high flexibility for different datasets. As it can be seen in Table III
, the ensemble members may vary, which suggests relatively high variance among databases in this field. Despite this variability, the performance of the ensemble still remained stable. In, the authors measured a human expert average false positive rate at the ROC dataset against the consensus of three human experts. This level is approximately 1 FP per image  for the ROC database, on which level our ensemble achieved the best score in the competition. Thus, we can recommend to use this level for thresholding at the ensemble creation phase and use it for detecting MAs on unknown images.
As for DR grading, our ensemble also performed well. It is also important to see how the different classes (R0, R1, R2, R3) are recognized at different levels. As it can be desired, the severity of DR affects the performance of our detector. At each threshold level, where the sensitivity is less than 1.0, the more severe case recognized with higher probability.
The selection of the appropriate threshold is also an important issue for our detector to provide sufficient sensitivity and specificity rate. In , the authors suggest that sensitivity is more important for a screening system than specificity. In opposition, the British Diabetic Association (BDA) recommends 80% sensitivity and 95% specificity for DR screening . In Table V, we can see that the most accurate result is achieved with the threshold value 0.9. By applying the first idea, we might consider the results corresponding to the threshold value 0.8 as the best in our experiment, where 96% sensitivity and 51% specificity are achieved. That is, we recognized almost all of the cases where DR is present, and half of the healthy ones. The closest to the second recommendation is the performance achieved at the 0.9 level: 76% sensitivity and 88% specificity.
It is difficult to compare our method to other screening systems. First of all, to the best of our knowledge, no other results reported for the complete Messidor database. Other screening systems are tested on private images. Unfortunately, the proportion of non-DR/DR cases are varying in these experiments. Abramoff et al.  reported 0.86 AUC on a population where 4.96% of the cases had at least minimum signs of DR. The databases on which Agurto et al.  tested, 74.43% and 76.26% cases contained signs of DR and they achieved 0.81 and 0.89 AUCs, respectively. The closest to match the requirements of BDA is the system of Jelinek et al.  with a 85% sensitivity and 90% specificity, where approximately 30% of patients had DR. Similar proportion (35.88%) of patients having DR are reported by Fleming et al.  in their automatic screening system.
Despite the promising results, our system still misclassifies some stage where serious case of DR is present. To improve grading performance, we must take into account the presence or absence of more DR-specific lesions (e.g. exudates), which are essential in a clinical setting. However, our MA detector can serve as a main component of such a system.
In this paper, we have proposed an ensemble-based microaneurysm detector which has proved its high efficiency in an open online challenge with its first position. Our novel framework relies on a set of preprocessing method, candidate extractor pairs, from which a search algorithm selects an optimal combination. Since our approach is modular, we can expect further improvements by adding more preprocessing methods and candidate extractors. We have also evaluated the grading performance of this detector in the 1200 images of the Messidor database. We have achieved a AUC value, which is competitive with the previously reported results on other databases. The grading results presented in this paper are already promising. However, a proper screening system should contain other components, which is expected to increase the performance of this approach, as well.
This work was supported in part by the János Bolyai grant of the Hungarian Academy of Sciences, and by the TECH08-2 project DRSCREEN - Developing a computer based image processing system for diabetic retinopathy screening of the National Office for Research and Technology of Hungary (contract no.: OM-00194/2008, OM-00195/2008, OM-00196/2008).
-  M. Abramoff, M. Niemeijer, M. Suttorp-Schulten, M. A. Viergever, S. R. Russel, and B. van Ginneken, “Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes,” Diabetes Care, vol. 31, pp. 193–198, 2008.
-  A. D. Fleming, K. A. Goatman, S. Philip, G. J. Prescott, P. F. Sharp, and J. A. Olson, “Automated grading for diabetic retinopathy: a large-scale audit using arbitration by clinical experts,” British Journal of Ophthalmology, vol. 94, no. 12, pp. 1606–1610, 2010.
-  H. J. Jelinek, M. J. Cree, D. Worsley, A. Luckie, and P. Nixon, “An automated microaneurysm detector as a tool for identification of diabetic retinopathy in rural optometric practice.” Clin Exp Optom, vol. 89, no. 5, pp. 299–305, 2006.
-  M. Abramoff, J. Reinhardt, S. Russell, J. Folk, V. Mahajan, M. Niemeijer, and G. Quellec, “Automated early detection of diabetic retinopathy,” Ophthalmology, vol. 117, no. 6, pp. 1147–1154, 2010.
-  M. Niemeijer, M. Loog, M. D. Abramoff, M. A. Viergever, M. Prokop, and B. van Ginneken, “On combining computer-aided detection systems,” IEEE Transactions on Medical Imaging, vol. 30, pp. 215 – 223, 2011.
-  B. Antal, I. Lazar, A. Hajdu, Z. Torok, A. Csutak, and T. Peto, “A multi-level ensemble-based system for detecting microaneurysms in fundus images,” in 4th IEEE International Workshop on Soft Computing Applications (SOFA 2010), 2010, pp. 137–142.
-  B. Antal and A. Hajdu, “Improving microaneurysm detection using an optimally selected subset of candidate extractors and preprocessing methods,” Pattern Recognition, vol. 45, no. 1, pp. 264 – 270, 2012.
-  A. A. A. Youssif, A. Z. Ghalwash, and A. S. Ghoneim, “Comparative study of contrast enhancement and illumination equalization methods for retinal vasculature segmentation,” Proc. Cairo International Biomedical Engineering Conferemce, 2006.
-  T. Walter and J. Klein, “Automatic detection of microaneyrysms in color fundus images of the human retina by means of the bounding box closing,” Lecture Notes in Computer Science, vol. 2526, pp. 210–220, 2002.
-  K. Zuiderveld, “Contrast limited adaptive histogram equalization,” Graphics gems, vol. IV, pp. 474–485, 1994.
S. Ravishankar, A. Jain, and A. Mittal, “Automated feature extraction for early detection of diabetic retinopathy in fundus images,” inComputer Vision and Pattern Recognition, 2009, pp. 210–217.
-  A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-based inpainting,” in Computer Vision and Pattern Recognition, vol. 2, 2003, pp. II–721 – II–728.
-  M. Niemeijer, B. van Ginneken, M. Cree, A. Mizutani, G. Quellec, C. Sanchez, B. Zhang, R. Hornero, M. Lamard, C. Muramatsu, X. Wu, G. Cazuguel, J. You, A. Mayo, Q. Li, Y. Hatanaka, B. Cochener, C. Roux, F. Karray, M. Garcia, H. Fujita, and M. Abramoff, “Retinopathy online challenge: Automatic detection of microaneurysms in digital color fundus photographs,” IEEE Transactions on Medical Imaging, vol. 29, no. 1, pp. 185–195, 2010.
-  T. Walter, P. Massin, A. Arginay, R. Ordonez, C. Jeulin, and J. C. Klein, “Automatic detection of microaneurysms in color fundus images,” Medical Image Analysis, vol. 11, pp. 555–566, 2007.
-  T. Spencer, J. A. Olson, K. C. McHardy, P. F. Sharp, and J. V. Forrester, “An image-processing strategy for the segmentation and quantification of microaneurysms in fluorescein angiograms of the ocular fundus,” Computers and Biomedical Research, vol. 29, pp. 284–302, 1996.
-  A. D. Fleming, S. Philip, and K. A. Goatman, “Automated microaneurysm detection using local contrast normalization and local vessel detection,” IEEE Transactions on Medical Imaging, vol. 25(9), pp. 1223–1232, 2006.
-  S. Abdelazeem, “Microaneurysm detection using vessels removal and circular hough transform,” Proceedings of the Nineteenth National Radio Science Conference, pp. 421 – 426, 2002.
-  B. Zhang, X. Wu, J. You, Q. Li, and F. Karray, “Detection of microaneurysms using multi-scale correlation coefficients,” Pattern Recogn., vol. 43, no. 6, pp. 2237–2248, 2010.
-  I. Lazar and A. Hajdu, “Microaneurysm detection in retinal images using a rotating cross-section based model,” in 2011 IEEE International Symposium on Biomedical Imaging, 2011, pp. 1405 –1409.
-  S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, vol. 220, pp. 671–680, 1983.
-  T. Kauppi, V. Kalesnykiene, J.-K. K m r inen, L. Lensu, I. Sorri, A. Raninen, R. Voutilainen, H. Uusitalo, H. K lvi inen, and J. Pietil , “Diaretdb1 diabetic retinopathy database and evaluation protocol,” Proc. of the 11th Conf. on Medical Image Understanding and Analysis (MIUA2007), pp. 61–65, 2007.
-  D. Chakraborty, “Clinical relevance of the roc and free-response paradigms for comparing imaging system efficacies,” Radiation Protection Dosimetry, vol. 139, no. 1-3, pp. 37–41, 2010.
-  J. Eng. Roc analysis: web-based calculator for roc curves. Johns Hopkins University, Baltimore. [Online]. Available: http://www.jrocfit.org
-  L. I. Kuncheva, Combining Pattern Classifiers. Methods and Algorithms. Wiley, 2004.
-  B. D. Association, “Retinal photography screening for diabetic eye disease,” Tech. Rep., 1997.
-  C. Agurto, E. S. Barriga, V. Murray, S. Nemeth, R. Crammer, W. Bauman, G. Zamora, M. S. Pattichis, and P. Soliz, “Automatic detection of diabetic retinopathy and age-related macular degeneration in digital fundus images,” Investigative Ophthalmology & Visual Science, vol. 52, no. 8, pp. 5862–5871, 2011.