1 Introduction
The work from Ho and Basu (2002) was seminal in analyzing the difficulty of a classification problem by using descriptors extracted from a learning dataset. Given that no Machine Learning (ML) technique can consistently obtain the best performance for every classification problem (Wolpert, 1996), this type of analysis allows to understand the scenarios in which a given ML technique succeeds and fails (Ali and Smith, 2006; Flores et al., 2014; Luengo and Herrera, 2015; Muñoz et al., 2018). Furthermore, it guides the development of new datadriven preprocessing and pattern recognition techniques, as done in (Dong and Kothari, 2003; Smith et al., 2014a; Mollineda et al., 2005; Hu et al., 2010; Garcia et al., 2015). This datadriven approach enables a better understanding of the peculiarities of a given application domain that can be explored in order to get better prediction results.
According to Ho and Basu (2002)
, the complexity of a classification problem can be attributed to a combination of three main factors: (i) the ambiguity of the classes; (ii) the sparsity and dimensionality of the data; and (iii) the complexity of the boundary separating the classes. The ambiguity of the classes is present in scenarios in which the classes can not be distinguished using the data provided, regardless of the classification algorithm employed. This is the case for poorly defined concepts and the use of nondiscriminative data features. These problems are known to have nonzero Bayes error. An incomplete or sparse dataset also hinders a proper data analysis. This shortage leads to some input space regions to be underconstrained. After training, subsequent data residing in those regions are classified arbitrarily. Finally,
Ho and Basu (2002) focus on the complexity of the classification boundary, and present a number of measures that characterize the boundary in different ways. The complexity of classification boundary is related to the size of the smallest description needed to represent the classes and is native of the problem itself (Macià, 2011). Using the Kolmogorov complexity concept (Ming and Vitanyi, 1993), the complexity of a classification problem can be measured by the size of the smallest algorithm which is able to describe the relationships between the data (Ling and AbuMostafa, 2006). In the worst case, it would be necessary to list all the objects along with their labels. However, if there is some regularity in the data, a compact algorithm can be obtained. In practice, the Kolmogorov complexity is incomputable and approximations are made, as those based on the computation of indicators and geometric descriptors drawn from the learning datasets available for training a classifer (Ho and Basu, 2002; Singh, 2003a). We refer to those indicators and geometric descriptors as data complexity measures or simply complexity measures from here on.This paper surveys the main complexity measures that can be obtained directly from the data available for learning. It extends the work from Ho and Basu (2002) by including more measures from literature that may complement the concepts already covered by the measures proposed in their work. The formulations of some of the measures are also adapted so that they can give standardized results. The usage of the complexity measures through recent literature is reviewed too, highlighting various domains where an advantageous use of the measures can be achieved. Besides, the main strengths and weakness of each measure are reported. As a side result, this analysis provides insights into adaptations needed with some of the measures, and into new unexplored areas where the complexity measures can succeed.
All measures detailed in this survey were assembled into an R package named ECoL (Extended Complexity Library). It contains all the measures from the DCoL (Data Complexity) library (OrriolsPuig et al., 2010), which were standardized and reimplemented in R, and a set of novel measures from the related literature. The added measures were chosen in order to complement the concepts assessed by the original complexity measures. Some corrections into the existent measures are also discussed and detailed in the paper. The ECoL package is publicly available at CRAN^{1}^{1}1https://cran.rproject.org/package=ECoL and GitHub^{2}^{2}2https://github.com/lpfgarcia/ECoL.
2 Complexity Measures
Geometric and statistical data descriptors are among the most used in the characterization of the complexity of classification problems. Among them are the measures proposed in (Ho and Basu, 2002) to describe the complexity of the boundary needed to separate binary classification problems, later extended to multiclass classification problems in works like (Mollineda et al., 2005; Ho et al., 2006; Mollineda et al., 2006; OrriolsPuig et al., 2010). Ho and Basu (2002) divide their measures into three main groups: (i) measures of overlap of individual feature values; (ii) measures of the separability of classes; and (iii) geometry, topology and density of manifolds measures. Similarly, Sotoca et al. (2005) divide the complexity measures into the categories: (i) measures of overlap; (ii) measures of class separability; and (iii) measures of geometry and density. In this paper, we group the complexity measures into more categories, as follows:

Featurebased measures, which characterize how informative the available features are to separate the classes;

Linearity measures, which try to quantify whether the classes can be linearly separated;

Neighborhood measures, which characterize the presence and density of same or different classes in local neighborhoods;

Network measures, which extract structural information from the dataset by modeling it as a graph;

Dimensionality measures, which evaluate data sparsity based on the number of samples relative to the data dimensionality;

Class imbalance measures, which consider the ratio of the numbers of examples between classes.
To define the measures, we consider that they are estimated from a learning dataset (or part of it) containing pairs of examples , where and . That is, each example is described by predictive features and has a label out of classes. Most of the measures are defined for features with numerical values only. In this case, symbolic values must be properly converted into numerical values prior to their use. We also use an assumption that linearly separable problems can be considered simpler than classification problems requiring nonlinear decision boundaries. Finally, some measures are defined for binary classification problems only. In that case, a multiclass problem must be first decomposed into multiple binary subproblems. Here we adopt a pairwise analysis of the classes, that is, a oneversusone (OVO) decomposition of the multiclass problem (Lorena et al., 2008). The measure for the multiclass problem is then defined as the average of the values across the different subproblems. In order to standardize the interpretation of the measure values, we introduce some modifications into the original definitions of some of the measures. The objective was to make each measure assume values in bounded intervals and also to make higher values of the measures indicative of a higher complexity, whilst lower values indicate a lower complexity.
2.1 Featurebased Measures
These measures evaluate the discriminative power of the features. In many of them each feature is evaluated individually. If there is at least one very discriminative feature in the dataset, the problem can be considered simpler than if there is no such an attribute. All measures from this category require the features to have numerical values. Most of the measures are also defined for binary classification problems only.
2.1.1 Maximum Fisher’s Discriminant Ratio (F1)
The first measure presented in this category is the maximum Fisher’s discriminant ratio, denoted by F1. It measures the overlap between the values of the features in different classes and is given by:
(1) 
where is a discriminant ratio for each feature . Originally, F1 takes the value of the largest discriminant ratio among all the available features. This is consistent with the definition that if at least one feature discriminates the classes, the dataset can be considered simpler than if no such attribute exists. In this paper we take the inverse of the original F1 formulation into account, as presented in Equation 1. Herewith, the F1 values become bounded in the interval and higher values indicate more complex problems, where no individual feature is able to discriminate the classes.
OrriolsPuig et al. (2010) present different equations for calculating , depending on the number of classes or whether the features are continuous or ordinal (Cummins, 2013). One straightforward formulation is:
(2) 
where is the proportion of examples in class , is the mean of feature across examples of class and
is the standard deviation of such values. An alternative for
computation which can be employed for both binary and multiclass classification problems is given in (Mollineda et al., 2005). Here we adopt this formulation, which is similar to the clustering validation index from Caliński and Harabasz (1974):(3) 
where is the number of examples in class , is the same as defined for Equation 2, is the mean of the values across all the classes, and denotes the individual value of the feature for an example from class . Taking, for instance, the dataset shown in Figure 1, the most discriminative feature would be . F1 correctly indicates that the classes can be easily separable using this feature. Feature
, on the other hand, is nondiscriminative, since its values for the two classes overlap, with the same mean and variance.
The denominator in Equation 3 must go through all examples in the dataset. The numerator goes through the classes. Since the discriminant ratio must be computed for all features, the total asymptotic cost for the F1 computation is . As (there is at least one example per class), can be reduced to .
Roughly, the F1 measure computes the ratio of interclass to the intraclass scatter for each feature. Using the formulation in Equation 1
, low values of the F1 measure indicate that there is at least one feature whose values show little overlap among the different classes; that is, it indicates the existence of a feature for which a hyperplane perpendicular to its axis can separate the classes fairly. Nonetheless, if the required hyperplane is oblique to the feature axes, F1 may not be able to reflect the underlying simplicity. In order to deal with this issue,
OrriolsPuig et al. (2010) propose to use a F1 variant based on aDirectional Vector
, to be discussed next. Finally, Hu et al. (2010)note that the F1 measure is most effective if the probability distributions of the classes are approximately normal, which is not always true. On the contrary, there can be highly separable classes, such as those distributed on the surfaces of two concentric hyperspheres, that would yield a very high value for F1.
2.1.2 The Directionalvector Maximum Fisher’s Discriminant Ratio (F1v)
This measure is used in OrriolsPuig et al. (2010) as a complement to F1. It searches for a vector which can separate the two classes after the examples have been projected into it and considers a directional Fisher criterion defined in Malina (2001) as:
(4) 
where is the directional vector onto which data are projected in order to maximize class separation, is the betweenclass scatter matrix and is the withinclass scatter matrix. , and are defined according to Equations 5, 6 and 7, respectively.
(5) 
where is the centroid (mean vector) of class and is the pseudoinverse of .
(6) 
(7) 
where is the proportion of examples in class and is the scatter matrix of class .
Taking the definition of dF, the F1v measure is given by:
(8) 
According to OrriolsPuig et al. (2010), the asymptotic cost of the F1v algorithm for a binary classification problem is . Multiclass problems are first decomposed according to the OVO strategy, producing subproblems. In the case that each one of them has the same number of examples, that is, , the total cost of the F1v measure computation is .
Lower values in F1v defined by Equation 8, which are bounded in the interval, indicate simpler classification problems. In this case, a linear hyperplane will be able to separate most if not all of the data, in a suitable orientation with regard to the features axes. This measure can be quite costly to compute due to the need for the pseudoinverse of the scatter matrix. Like F1, it is based on the assumption of normality of the classes distributions.
2.1.3 Volume of Overlapping Region (F2)
The F2 measure calculates the overlap of the distributions of the features values within the classes. It can be determined by finding, for each feature , its minimum and maximum values in the classes. The range of the overlapping interval is then calculated, normalized by the range of the values in both classes. Finally, the obtained values are multiplied, as shown in Equation 9.
(9) 
where:
The values and are the maximum and minimum values of each feature in a class , respectively. The numerator becomes zero when the perclass value ranges are disjoint for at least one feature. This equation uses a correction that was made in Souto et al. (2010) and Cummins (2013) to the original definition of F2, which may yield negative values for nonoverlapping feature ranges. The asymptotic cost of this measure is , considering a OVO decomposition in the case of multiclass problems. The higher the F2 value, the greater the amount of overlap between the problem classes. Therefore, the problem’s complexity is also higher. Moreover, if there is at least one nonoverlapping feature, the F2 value should be zero. Figure 2 illustrates the region that F2 tries to capture (as the shaded area), for a dataset with two features and two classes.
Cummins (2013) points to an issue with F2 for the cases illustrated in Figure 3. In Figure (a)a, the attribute is discriminative but the minimum and maximum values overlap in the different classes; and in Figure (b)b, there is one noisy example which disrupts the measure values. Cummins (2013) proposes to deal with these situations by counting the number of feature values in which there is an overlap, which is only suitable for discretevalued features. Using this solution, continuous features must be discretized a priori, which imposes the difficulty of choosing a proper discretization technique and associated parameters, an open issue in data mining (Kotsiantis and Kanellopoulos, 2006).
It should be noted that the situations shown in Figure 3 can be also harmful for the F1 measure. As noted by Hu et al. (2010), F2 does not capture the simplicity of a linear oblique border either, since it assumes again that the linear boundary is perpendicular to one of the features axes. Finally, the F2 value can become very small depending on the number of operands in Equation 9. That is, it is highly dependent on the number of features a dataset has. This worsens for problems with many features, so that their F2 values may not be comparable to those of other problems with fewer features. Souto et al. (2010), Lorena et al. (2012) and, more recently, SeijoPardo et al. (2019) use a sum instead of the product in Equation 9, which partially solves the problems identified. Nonetheless, the result is not an overlapping volume and corresponds to the amount or size of the overlapping region. In addition, the measure remains influenced by the number of features the dataset has.
2.1.4 Maximum Individual Feature Efficiency (F3)
This measure estimates the individual efficiency of each feature in separating the classes, and considers the maximum value found among the features. Here we take the complement of this measure so that higher values are obtained for more complex problems. For each feature, it checks whether there is overlap of values between examples of different classes. If there is overlap, the classes are considered to be ambiguous in this region. The problem can be considered simpler if there is at least one feature which shows low ambiguity between the classes, so F3 can be expressed as:
(10) 
where gives the number of examples that are in the overlapping region for feature and can be expressed by Equation 11. Low values of F3, computed by Equation 10, indicate simpler problems, where few examples overlap in at least one dimension. As with F2, the asymptotic cost of the F3 measure is .
(11) 
In Equation 11, is the indicator function, which returns 1 if its argument is true and 0 otherwise, while and are the same as defined for F2.
Figure 4 depicts the computation of F3 for the same dataset from Figure 2. While for feature the proportion of examples that are in the overlapping region is (Figure (a)a), for this proportion is (Figure (b)b), resulting in a F3 value of .
Since is calculated taking into account the minimum and maximum values of the feature in different classes, it entails the same problems identified for F2 with respect to: classes in which the feature has more than one valid interval (Figure (a)a), susceptibility to noise (Figure (b)b) and the fact that it is assumed that in linearly separable problems, the boundary is perpendicular to an input axis.
2.1.5 Collective Feature Efficiency (F4)
The F4 measure was proposed in OrriolsPuig et al. (2010) to get an overview of how the features work together. It successively applies a procedure similar to that adopted for F3. First the most discriminative feature according to F3 is selected; that is, the feature which shows less overlap between different classes. All examples that can be separated by this feature are removed from the dataset and the previous procedure is repeated: the next most discriminative feature is selected, excluding the examples already discriminated. This procedure is applied until all the features have been considered and can also be stopped when no example remains. F4 considers the ratio of examples that have not been discriminated, as presented in Equation 12. F4 is computed after rounds are performed through the dataset, where is in the range . If one of the input features is already able to discriminate all the examples in , is 1, whilst it can get up to in the case all features have to be considered. Its equation can be denoted by:
(12) 
where measures the number of points in the overlapping region of feature for the dataset from the th round (). This is the current most discriminative feature in . Taking the th iteration of F4, the most discriminative feature in dataset can be found using Equation 13, adapted from F3.
(13) 
where is computed according to Equation 11. While the dataset at each round can be defined as:
(14)  
(15) 
That is, the dataset at the th round is reduced by removing all examples that are already discriminated by the previous considered feature . Therefore, the computation of F4 is similar to that of F3, except that it can be applied to reduced datasets. Lower values of F4 computed by Equation 12 indicate that it is possible to discriminate more examples and, therefore, that the problem is simpler. The idea is to get the number of examples that can be correctly classified if hyperplanes perpendicular to the axes of the features are used in their separation. Since the overlapping measure applied is similar to that used for F3, they share the same problems in some estimates (as discussed for Figures (a)a and (b)b). F4 applies the F3 measure multiple times and at most it will iterate for all input features, resulting in a worst case asymptotic cost of .
Figure 5 shows the F4 operation for the dataset from Figure 2. Feature is the most discriminative in the first round (Figure (a)a). Figure (a)a shows the resulting dataset after all examples correctly discriminated by are disregarded. Figure (c)c shows the final dataset after feature has been analyzed in Figure (b)b. The F4 value for this dataset is .
2.2 Measures of Linearity
These measures try to quantify to what extent the classes are linearly separable, that is, if it is possible to separate the classes by a hyperplane. They are motivated by the assumption that a linearly separable problem can be considered simpler than a problem requiring a nonlinear decision boundary. To obtain the linear classifier, Ho and Basu (2002) suggest to solve an optimization problem proposed by Smith (1968), while in OrriolsPuig et al. (2010)
a linear Support Vector Machine (SVM)
(Cristianini and ShaweTaylor, 2000) is used instead. Here we adopt the SVM solution.The hyperplane sought in the SVM formulation is the one which separates the examples from different classes with a maximum margin while minimizing training errors. This hyperplane is obtained by solving the following optimization problem:
(16) 
(17) 
where is the tradeoff between the margin maximization, achieved by minimizing the norm of , and the minimization of the training errors, modeled by . The hyperplane is given by , where is a weight vector and is an offset value. SVMs are originally proposed to solve binary classification problems with numerical features. Therefore, symbolic features must be converted into numerical values and multiclass problems must be first decomposed.
2.2.1 Sum of the Error Distance by Linear Programming (L1)
This measure assesses if the data are linearly separable by computing, for a dataset, the sum of the distances of incorrectly classified examples to a linear boundary used in their classification. If the value of L1 is zero, then the problem is linearly separable and can be considered simpler than a problem for which a nonlinear boundary is required.
Given the SVM hyperplane, the error distance of the erroneous instances can be computed by summing up the values. For examples correctly classified, will be zero, whist it indicates the distance of the example to the linear boundary otherwise. This is expressed in Equation 18. The values are determined in the SVM optimization process.
(18) 
The L1 value can then be computed as:
(19) 
Low values for L1 (bounded in ) indicate that the problem is close to being linearly separable, that is, simpler. Figure 6 illustrates an example of L1 application. After a linear boundary is obtained, the values of the misclassified examples (gray circles) are summed up and subject to Equation 19.
L1 does not allow to check if a linearly separable problem is simpler than another that is also linearly separable. Therefore, a dataset for which data are distributed narrowly along the linear boundary will have a null L1 value, and so will a dataset in which the classes are far apart with a large margin of separation. The asymptotic computing cost of the measure is dependent on that of the linear SVM, and can take operations in the worst case (Bottou and Lin, 2007). In multiclass classification problems decomposed according to OVO, this cost would be , which resumes to too.
2.2.2 Error Rate of Linear Classifier (L2)
The L2 measure computes the error rate of the linear SVM classifier. Let denote the linear classifier obtained. L2 is then given by:
(20) 
Higher L2 values denote more errors and therefore a greater complexity regarding the aspect that the data cannot be separated linearly. For the dataset in Figure 6, the L2 value is . L2 has similar issues with L1 in that it does not differentiate between problems that are barely linearly separable (i.e., with a narrow margin) from those with classes that are very far apart. The asymptotic cost of L2 is the same of L1, that is, .
2.2.3 NonLinearity of a Linear Classifier (L3)
This measure uses a methodology proposed by Hoekstra and Duin (1996)
. It first creates a new dataset by interpolating pairs of training examples of the same class. Herewith, two examples from the same class are chosen randomly and they are linearly interpolated (with random coefficients), producing a new example. Figure
7 illustrates the generation of six new examples (in gray) from a base training dataset. Then a linear classifier is trained on the original data and has its error rate measured in the new data points. This index is sensitive to how the data from a class are distributed in the border regions and also on how much the convex hulls which delimit the classes overlap. In particular, it detects the presence of concavities in the class boundaries (Armano and Tamponi, 2016). Higher values indicate a greater complexity. Letting denote the linear classifier induced from the original training data , the L3 measure can be expressed by:(21) 
where is the number of interpolated examples and their corresponding labels are denoted by . In ECoL we generate the interpolated examples maintaining the proportion of examples per class from the original dataset and use . The asymptotic cost of this measure is dependent on both the induction of a linear SVM and the time taken to obtain the predictions for the test examples, resulting in .
2.3 Neighborhood Measures
These measures try to capture the shape of the decision boundary and characterize the class overlap by analyzing local neighborhoods of the data points. Some of them also capture the internal structure of the classes. All of them work over a distance matrix storing the distances between all pairs of points in the dataset. To deal with both symbolic and numerical features, we adopt a heterogeneous distance measure named Gower (Gower, 1971). For symbolic features, the Gower metric computes if the compared values are equal, whilst for numerical features, a normalized difference of values is taken.
2.3.1 Fraction of Borderline Points (N1)
In this measure, first a Minimum Spanning Tree (MST) is built from data, as illustrated in Figure 8. Herewith, each vertex corresponds to an example and the edges are weighted according to the distance between them. N1 is obtained by computing the percentage of vertices incident to edges connecting examples of opposite classes in the generated MST. These examples are either on the border or in overlapping areas between the classes. They can also be noisy examples surrounded by examples from another class. Therefore, N1 estimates the size and complexity of the required decision boundary through the identification of the critical points in the dataset: those very close to each other but belong to different classes. Higher N1 values indicate the need for more complex boundaries to separate the classes and/or that there is a large amount of overlapping between the classes. N1 can be expressed as:
(22) 
To build the graph from the data, it is necessary to first compute the distance matrix between all pairs of elements, which requires operations. Next, using Prim’s algorithm for obtaining the MST requires operations in the worst case. Therefore, the total asymptotic complexity of N1 is .
N1 is sensitive to the type of noise where the closest neighbors of noisy examples have a different class from their own, as typical in the scenario where erroneous class labels are introduced during data preparation. Datasets with this type of noise are considered more complex than their clean counterparts, according to the N1 measure, as observed in Lorena et al. (2012) and Garcia et al. (2015).
Another issue is that there can be multiple MSTs valid for the same set of points. Cummins (2013) propose to generate ten MSTs by presenting the data points in different orderings and reporting an average N1 value. Basu and Ho (2006) also report that the N1 value can be large even for a linearly separable problem. This happens when the distances between borderline examples are smaller than the distances between examples from the same class. On the other hand, Ho (2002) suggests that a problem with a complicated nonlinear class boundary can still have relatively few edges among examples from different classes as long as the data points are compact within each class.
2.3.2 Ratio of Intra/Extra Class Nearest Neighbor Distance (N2)
This measure computes the ratio of two sums: (i) the sum of the distances between each example and its closest neighbor from the same class (intraclass); and (ii) the sum of the distances between each example and its closest neighbor from another class (extraclass). This is shown in Equation 24.
(23) 
where corresponds to the distance of example to its nearest neighbor (NN) from its own class and represents the distance of to the closest neighbor from another class (’s nearest enemy). Based on the intra/extra class calculation, N2 can be obtained as:
(24) 
The computation of N2 requires obtaining the distance matrix between all pairs of elements in the dataset, which requires operations. Figure 9 illustrates the intra and extraclass distances for a particular example in a dataset.
Low N2 values are indicative of simpler problems, in which the overall distance between examples of different classes exceeds the overall distance between examples from the same class. N2 is sensitive to how data are distributed within classes and not only to how the boundary between the classes is like. It can also be sensitive to labeling noise in the data, just like N1. According to Ho (2002), a high N2 value can also be obtained for a linearly separable problem where the classes are distributed in a long, thin, and sparse structure along the boundary. It must be also observed that N2 is related to F1 and F1v, since they all assess intra and inter class variabilities. However, N2 uses a distance that summarizes the joint relationship between the values of all the features for the concerned examples.
2.3.3 Error Rate of the Nearest Neighbor Classifier (N3)
The N3 measure refers to the error rate of a 1NN classifier that is estimated using a leaveoneout procedure. The following equation denotes this measure:
(25) 
where represents the nearest neighbor classifier’s prediction for example using all the others as training points. High N3 values indicate that many examples are close to examples of other classes, making the problem more complex. N3 requires operations.
2.3.4 NonLinearity of the Nearest Neighbor Classifier (N4)
This measure is similar to L3, but uses the NN classifier instead of the linear predictor. It can be expressed as:
(26) 
where is the number of interpolated points, generated as illustrated in Figure 7. Higher N4 values are indicative of problems of greater complexity. In contrast to L3, N4 can be applied directly to multiclass classification problems, without the need to decompose them into binary subproblems first. The asymptotic cost of computing N4 is operations, as it is necessary to compute the distances between all possible testing and training examples.
2.3.5 Fraction of Hyperspheres Covering Data (T1)
This is regarded as a topological measure in Ho and Basu (2002). It uses a process that builds hyperspheres centered at each one of the examples. The radius of each hypersphere is progressively increased until the hypersphere reaches an example of another class. Smaller hyperspheres contained in larger hyperspheres are eliminated. T1 is defined as the ratio between the number of the remaining hyperspheres and the total number of examples in the dataset:
(27) 
where gives the number of hyperspheres that are needed to cover the dataset.
The hyperspheres represent a form of adherence subsets as discussed in Lebourgeois and Emptoz (1996). The idea is to obtain an adherence subset of maximum order for each example such that it includes only examples from the same class. Subsets that are completely included in other subsets are discarded. In principle the adherence subsets can be of any form (e.g. hyperectangular), and hyperspheres are chosen in the definition of this measure because it can be defined with relatively few parameters (i.e., only a center and a radius). Fewer hyperspheres are obtained for simpler datasets. This happens when data from the same class are densely distributed and close together. Herewith, this measure also captures the distribution of data within the classes and not only their distribution near the class boundary.
In this paper we propose an alternative implementation of T1. It involves a modification of the definition to stop the growth of the hypersphere when the hyperspheres centered at two points of opposite classes just start to touch. With this modification, the radius of each hypersphere around an example can be directly determined based on distance matrix between all examples. The radius computation for an example is shown in Algorithm 1, in which the nearest enemy () of an example corresponds to the nearest data point from an opposite class (). If two points are mutually nearest enemies of each other (line 3 in Algorithm 1), the radiuses of their hyperspheres correspond to half of the distance between them (lines 4 and 5, also see Figure (a)a). The radiuses of the hyperspheres around other examples can be determined recursively (lines 7 and 8), as illustrated in Figure (b)b.
Once the radiuses of all hyperspheres are found, a postprocessing step can be applied to verify which hyperspheres are absorbed: those lying inside larger hyperspheres. The hyperspheres obtained for our example dataset is shown in Figure (c)c. The most demanding operation in T1 is to compute the distance matrix between all the examples in the dataset, which requires operations.
2.3.6 Local Set Average Cardinality (LSC)
According to Leyva et al. (2014), the LocalSet (LS) of an example in a dataset () is defined as the set of points from whose distance to is smaller than the distance from to ’s nearest enemy (Equation 28).
(28) 
where is the nearest enemy from example . Figure 11 illustrates the local set of a particular example (, in gray) in a dataset.
The cardinality of the LS of an example indicates its proximity to the decision boundary and also the narrowness of the gap between the classes. Therefore, the LS cardinality will be lower for examples separated from the other class with a narrow margin. According to Leyva et al. (2014), a high number of lowcardinality local sets in a dataset suggests that the space between classes is narrow and irregular, that is, the boundary is more complex. The local set average cardinality measure (LSC) is calculated here as:
(29) 
where is the cardinality of the local set for example . This measure can complement N1 and L1 by also revealing the narrowness of the betweenclass margin. Higher values are expected for more complex datasets, in which each example is nearest to an enemy than to other examples from the same class. In that case, each example will have a local set of cardinality 1, resulting in a LSC of . The asymptotic cost of LSC is dominated by the computation of pairwise distances between all examples, resulting in operations.
2.4 Network Measures
Morais and Prati (2013) and Garcia et al. (2015) model the dataset as a graph and extract measures for the statistical characterization of complex networks Kolaczyk (2009) from this representation. In Garcia et al. (2015) low correlation values were observed between the basic complexity measures of Ho and Basu (2002)
and the graphbased measures, which supports the relevance of exploring this alternative representation of the data structure. In this paper, we highlight the best measures for the data complexity induced by label noise imputation (
Garcia et al. (2015)), with an emphasis on those with low correlation between each other.To use these measures, it is necessary to represent the classification dataset as a graph. The obtained graph must preserve the similarities or distances between examples for modeling the data relationships. Each example from the dataset corresponds to a node or vertex of the graph, whilst undirected edges connect pairs of examples and are weighted by the distances between the examples. As in the neighborhood measures, the Gower distance is employed. Two nodes and are connected only if . This corresponds to the NN method for building a graph from a dataset in the attributevalue format (Zhu et al., 2005). As in Morais and Prati (2013) and Garcia et al. (2015), in ECoL the value is set to 0.15 (note that the Gower distance is normalized to the range [0,1]). Next, a postprocessing step is applied to the graph, pruning edges between examples of different classes. Figure 12 illustrates the graph building process for the dataset from Figure 2. Figure (a)a shows the first step, when the pairs of vertices with are connected. This first step is unsupervised, since it disregards the labels of connected points. Figure (b)b shows the graph obtained after the pruning process is applied to disconnect examples from different classes. This step can be regarded as supervised, in which the label information is taken into account to obtain the final graph.
For a given dataset, let denote the graph built by this process. By construction, and . Let the th vertex of the graph be denoted as and an edge between two vertices and be denoted as . The extracted measures are described next. All the measures from this category require building a graph based on the distance matrix between all pairs of elements, which requires operations. The asymptotic cost of all the presented measures is dominated by the computation of this matrix.
2.4.1 Average density of the network (Density)
This measure considers the number of edges that are retained in the graph built from the dataset normalized by the maximum number of edges between pairs of data points.
(30) 
Lower values for this measure are obtained for dense graphs, in which many examples get connected. This will be the case for datasets with dense regions from a same class in the dataset. This type of dataset can be regarded as having lower complexity. On the other hand, a low number of edges will be observed for datasets of low density (examples are far apart in the input space) and/or for which examples of opposite classes are near each other, implying a higher classification complexity.
2.4.2 Clustering coefficient (ClsCoef)
The clustering coefficient of a vertex is given by the ratio of the number of edges between its neighbors and the maximum number of edges that could possibly exist between them. We take as a complexity measure the value:
(31) 
where denotes the neighborhood set of a vertex (those nodes directly connected to ) and is the size of . The sum calculates, for each vertex , the ratio of existent edges between its neighbors by the total number of edges that could possibly be formed.
The Clustering coefficient measure assesses the grouping tendency of the graph vertexes, by monitoring how close to form cliques neighborhood vertexes are. As computed by Equation 31, it will be smaller for simpler datasets, which will tend to have dense connections among examples from the same class.
2.4.3 Hub score (Hubs)
The hub score scores each node by the number of connections it has to other nodes, weighted by the number of connections these neighbors have. Herewith, highly connected vertexes which are also connected to highly connected vertices will have a larger hub score. This is a measure of the influence of each node of the graph. Here we take the formulation:
(32) 
The values of
are given by the principal eigenvector of
, where is the adjacency matrix of the graph. Here, we take an average value for all vertices.In complex datasets, in which a high overlapping of the classes is observed, strong vertexes will tend to be less connected to strong neighbors. On the other hand, for simple datasets there will be dense regions within the classes and higher hub scores. Therefore, according to Equation 32 smaller Hubs values are expected for simpler datasets.
2.5 Dimensionality Measures
The measures from this category give an indicative of data sparsity. They are based on the dimensionality of the datasets, either original or reduced. The idea is that it can be more difficult to extract good models from sparse datasets, due to the probable presence of regions of low density that will be arbitrarily classified.
2.5.1 Average number of features per dimension (T2)
Originally, T2 divides the number of examples in the dataset by their dimensionality (Basu and Ho, 2006). In this paper we take the inverse of this formulation in order to obtain higher values for more complex datasets, so that:
(33) 
T2 can be computed at . In some work the logarithmic function is applied to the measure (ex. Lorena et al. (2012)) because T2 can take arbitrarily large or small values. Though, this can take the measure into negative values when the number of examples is larger than the number of features.
T2 reflects the data sparsity. If there are many predictive attributes and few data points, they will be probably sparsely distributed in the input space. The presence of low density regions will hinder the induction of an adequate classification model. Therefore, lower T2 values indicate less sparsity and therefore simpler problems.
2.5.2 Average number of PCA dimensions per points (T3)
The measure T3 (Lorena et al., 2012)
is defined with a Principal Component Analysis (PCA) of the dataset. Instead of the raw dimensionality of the feature vector (as in T2), T3 uses the number of PCA components needed to represent 95% of data variability (
) as the base of data sparsity assessment. The measure is calculated as:(34) 
The value can be regarded as an estimate of the intrinsic dataset dimensionality after the correlation among features is minimized. As in the case of T2, smaller values will be obtained for simpler datasets, which will be less sparse. Since this measure requires performing a PCA analysis of the dataset, its worst cost is .
2.5.3 Ratio of the PCA Dimension to the Original Dimension (T4)
This measure gives a rough measure of the proportion of relevant dimensions for the dataset (Lorena et al., 2012). This relevance is measured according to the PCA criterion, which seeks a transformation of the features to uncorrelated linear functions of them that are able to describe most of the data variability. T4 can be expressed by:
(35) 
The larger the T4 value, the more of the original features are needed to describe data variability. This indicates a more complex relationship of the input variables. The asymptotic cost of the measure is .
2.6 Class Imbalance Measures
These measures try to capture one aspect that may largely influence the predictive performance of ML techniques when solving data classification problems: class imbalance, that is, a large difference in the number of examples per class in the training dataset. Indeed, when the differences are severe, most of the ML classification techniques tend to favor the majority class and present generalization problems.
In this section we present some measures for capturing class imbalance. If the problem has a high imbalance in the proportion of examples per class, it can be considered more complex than a problem for which the proportions are similar.
2.6.1 Entropy of class proportions (C1)
The C1 measure was used in Lorena et al. (2012) to capture the imbalance in a dataset. It can be expressed as:
(36) 
where is the proportion of examples in each of the classes. This measure will achieve maximum value for balanced problems, that is, problems in which all proportions are equal. These can be considered simpler problems according to the class balance aspect. The asymptotic cost for computing this measure is for obtaining the proportions of examples per class.
2.6.2 Imbalance ratio (C2)
The C2 measure is a well known index computed for measuring class balance. Here we adopt a version of the measure that is also suited for multiclass classification problems (Tanwani and Farooq, 2010):
(37) 
where:
(38) 
where is the number of instances from the th class. These numbers can be computed at operations. Larger values of C2 are obtained for imbalanced problems. The minimum value of C2 is achieved for balanced problems, in which for all .
2.7 Other Measures
This section gives an overview of some other measures that can be used to characterize the complexity of classification problems found in the relate literature. Part of these measures was not formally included previously because they capture similar aspects already measured by the described measures. Other measures were excluded because they have a high computational cost.
Walt and Barnard (2007) present some variations of the T1 measure. One of them is quite similar to the LSC measure, with a difference on the normalization used by LSC. Another variation first generates an MST connecting the hyperspheres centers given by T1 and then counts the number of vertexes that connect examples from different classes. There is also a measure that computes the density of the hyperspheres. We believe that the LSC measure complements T1 at a lower computational cost.
Mollineda et al. (2006) present some density measures. The first one, named D1, gives the average number of examples per unit of volume in the dataset. The volume of local neighborhood (D2) measure gives the average volume occupied by the nearest neighbors of each example. Finally, the class density in overlap region (D3) determines the density of each class in the overlap regions. It counts, for each class, the number of points lying in the same region of a different class. Although these measures give an overview of data density, we believe that they do not allow to extract complementary views of the problem complexity already captured by the original neighborhoodbased measures. Furthermore, they may have a higher computational cost and present an additional parameter (e.g. the in nearest neighbors) to be tuned.
Some of the measures found in the literature propose to analyze the dataset using a divisive approach or in multiple resolutions. Usually they show a high computational cost, that can be prohibitive for datasets with a moderate number of features. Singh (2003a) reports some of such measures. Their partitioning algorithm generates hypercuboids in the space, at different resolutions (with increasing numbers of intervals per feature from 0 to 31). At each resolution, the data points are assigned into cells. Purity measures whether the cells contain examples from a same class or from mixed classes. The nearest neighbor separability measure counts, for each example of a cell, the proportion of its nearest neighbors that share its class. The cell measurements are linearly weighted to obtain a single estimate and the overall measurement across all cells at a given resolution is exponentially weighted. Afterwards, the area under the curve defined by one separability measure versus the resolution defines the overall data separability. In Singh (2003b) two more measures based on the space partitioning algorithm are defined: collective entropy, which is the level of uncertainty accumulated at different resolutions; and data compactness, related to the proportion of nonempty cells at different resolutions.
In Armano and Tamponi (2016) a method named Multiresolution Complexity Analysis (MRCA) is used to partition a dataset. Like in T1, hyperspheres of different amplitudes are drawn around the examples and the imbalance regarding how many examples of different classes they contain is measured. A new dataset of profile patterns is obtained, which is clustered. Afterwards, each cluster is evaluated and ranked according to a complexity metric called Multiresolution Index (MRI).
Armano (2015) presents how to obtain a class signature which can be used to identify, for instance, the discriminative capability of the input features. This could be regarded as a featurebased complexity measure, although more developments are necessary, since the initial studies considered binaryvalued features only.
Mthembu and Marwala (2008) present a Separability Index SI, which takes into account the average number of examples in a dataset that have a nearest neighbor with the same label. This is quite similar to what is captured by N3, except for using more neighbors in NN classification. Another measure named Hypothesis margin (HM) takes the distance between the nearest neighbor of an object of the same class and a nearest enemy of another class. This largely resembles the N2 computation.
Similarly to D3, Mollineda et al. (2006) and Anwar et al. (2014) introduce a complexity measure which also focuses on local information for each example by employing the nearest neighbor algorithm. If the majority of the nearest neighbors of an example share its label, this point can be regarded as easy to classify. Otherwise, it is a difficult point. An overall complexity measure is given by the proportion of data points classified as difficult.
Leyva et al. (2014) define some measures based on the concept of Local Sets previously described, which employ neighborhood information. Besides LSC, Leyva et al. (2014) also propose to cluster the data in the local sets and then count the number of obtained clusters. This measure is related to T1. The third measure is named number of invasive points (Ipoints), which uses the local sets to identify borderline instances and is related to N1, N2 and N3.
Smith et al. (2014a) propose a set of measures devoted to understand why some data points are harder to classify than others. They are called instance hardness measures. One advantage of such approach is to reveal the difficulty of a problem at the instance level, rather than at the aggregate level with the entire dataset. Nonetheless, the measures can be averaged to give an estimate at the dataset level. As shown in some recent work on dynamic classifier selection (Cruz et al., 2017b, 2018), the concept of instance hardness is very useful for pointing out classifiers able to perform well in confusing or overlapping areas of the dataset, giving indicatives of a local level of competence. Most of the complexity measures previously presented, although formulated for obtaining a complexity estimate per dataset, can be adapted in order to assess the contribution of each example to the overall problem difficulty. Nonetheless, this is beyond the scope of this paper.
One very effective instance hardness measure from (Smith et al., 2014a) is the Disagreeing Neighbors (DN), which gives the percentage of the nearest neighbors that do not share the label of an example. This same concept was already explored in the works of Sotoca et al. (2005); Mthembu and Marwala (2008); Anwar et al. (2014). The Disjunct Size (DS) corresponds to the size of a disjunct that covers an example divided by the largest disjunct produced, in which disjuncts are obtained using the C4.5 learning algorithm. A relate measure is the Disjunct Class Percentage (DCP), which is the number of data points in a disjunct that belong to a same class divided by the total number of examples in the disjunct. The Tree Depth
(TD) returns the depth of the leaf node that classifies an instance in a decision tree. The previous measures give estimates from the perspective of a decision tree classifier. In addition, the
Minority Value (MV) index is the ratio of examples sharing the same label of an example to the number of examples in the majority class. The Class Balance(CB) index presents an alternative to measuring the class skew. The C1 and C2 measures previously described are simple alternatives already able to capture the class imbalance aspect.
Elizondo et al. (2012) focus their study on the relationship between linear separability and the level of complexity of classification datasets. Their method uses
Recursive Deterministic Perceptron
(RDP) models and counts the number of hyperplanes needed to transform the original problem, which may not be linearly separable, into a linearly separable problem.In Skrypnyk (2011)
various class separability measures are presented, focusing on feature selection. Some parametric measures are the
Mahalanobis and the Bhattacharyya distances between the classes and the Normal Information Radius. These measures are computationally intensive due to the need to compute covariance matrices and their inverse. An information theoretic measure is the KullbackLeibler distance. It quantifies the discrepancy between two probability distributions. Based on discriminant analysis, a number of class separability measures can also be defined. This family of techniques is closely related to measures F1v and N2 discussed in this survey.Cummins (2013) also defines some alternative complexity measures. The first, named N5, consists of multiplying N1 by N2. According to Fornells et al. (2007), the multiplication of N1 and N2 emphasizes extreme behavior concerning class separability. Another measure (named Case Base Complexity Profile) retrieves the nearest neighbors of an example for increasing values of , from 1 up to a limit . At each round, the proportion of neighbors that have the same label as is counted. The obtained values are then averaged. Although interesting, this measure can be considered quite costly to compute.
More recently, Zubek and Plewczynski (2016) presented a complexity curve based on the Hellinger distance of probability distributions, assuming that the input features are independent. It takes subsets of different sizes from a dataset and verifies if their information content is similar to that of the original dataset. The computed values are plotted and the area under the obtained curve is used as an estimate of data complexity. The proposed measure is also applied in data pruning. The measure values computed turned out to be quite correlated to T2.
In the recent literature there are also studies on generalizations of the complexity measures for other types of problems. In Lorena et al. (2018) these measures are adapted to quantify the difficulty of regression problems. Charte et al. (2016) present a complexity score for multilabel classification problems. SmithMiles (2009) surveys some strategies for measuring the difficulty of optimization problems.
3 The ECol Package
Based on the review performed, we assembled a set of 22 complexity measures into an R package named ECoL (Extended Complexity Library), available at CRAN^{3}^{3}3https://cran.rproject.org/package=ECoL and GitHub^{4}^{4}4https://github.com/lpfgarcia/ECoL. Table 1 summarizes the characteristics of the complexity measures included in the package. It presents the category, name, acronym, and the limit values (minimum and maximum) assumed by these measures. Taking the measure F1 as an example, according to Table 1 its higher limit is 1 (when the average values of the attributes are the same for all classes), its lower value is approximately null. In our implementations, all measures assume values that are in bounded intervals. Moreover, for all measures, the higher the value, the greater the complexity measured. We also present the worst case asymptotic time complexity cost for computing the measures, where stands for the number of points in a dataset, corresponds to its number of features, is the number of classes and is the number of novel points generated in the case of the measures L3 and N4. All distance and networkbased measures are based on information from a distance matrix between all pairs of examples of the dataset, which can be computed only once and reused for obtaining the values of all those measures. The same reasoning applies to the linearity measures, since all of them involve training a linear SVM, from which the required information for computing the individual measures can be obtained.
Category  Name  Acronym  Min  Max  Asymptotic cost 
Featurebased  Maximum Fisher’s discriminant ratio  F1  1  
Directional vector maximum Fisher’s discriminant ratio  F1v  1  
Volume of overlapping region  F2  
Maximum individual feature efficiency  F3  
Collective feature efficiency  F4  
Linearity  Sum of the error distance by linear programming 
L1  
Error rate of linear classifier  L2  
Non linearity of linear classifier  L3  
Neighborhood  Faction of borderline points  N1  
Ratio of intra/extra class NN distance  N2  
Error rate of NN classifier  N3  
Non linearity of NN classifier  N4  
Fraction of hyperspheres covering data  T1  
Local set average cardinality  LSC  0  
Network  Density  Density  
Clustering Coefficient  ClsCoef  
Hubs  Hubs  
Dimensionality  Average number of features per dimension  T2  
Average number of PCA dimensions per points  T3  
Ratio of the PCA dimension to the original dimension  T4  0  1  
Class imbalance  Entropy of classes proportions  C1  0  1  
Imbalance ratio  C2  0  1 
Another relevant observation is that although each measure gives an indication into the complexity of the problem according to some characteristics of its learning dataset, a unified interpretation of their values is not easy. Each measurement has an associated limitation (for example, the feature separability measures cannot cope with situations where an attribute has different ranges of values for the same class  Figure (a)a) and must then be considered only as an estimate of the problem complexity, which may have associated errors. Since the measures are made on a dataset , they also give only an apparent measurement of the problem complexity (Ho and Basu, 2002). This reinforces the need to analyze the measures together to provide more robustness to the reached conclusions. There are also cases where some caution must be taken, such as in the case of F2, whose final values depend on the number of predictive attributes in the dataset. This particular issue is pointed out by Singh (2003b), which states that the complexity measures ideally should be conceptually uncorrelated to the number of features, classes, or number of data points a dataset has, making the complexity measure values for different datasets more comparable. This requirement is clearly not fulfilled by F2. Nonetheless, for the dimensionality and balancebased measures, Singh (2003a)’s assertion does not apply, since they are indeed concerned with the relationship of the numbers of dimensions and data points a dataset has.
For instance, a linearly separable problem with an oblique hyperplane will have high F1, indicating that it is complex, and also a low L1, denoting that it is simple. LSC, on the other hand, will assume a low value for a very imbalanced twoclass dataset in which one of the classes contains one unique example and the other class is far and densely distributed. This would be an indicative of a simple classification problem according to LSC interpretation, but data imbalance should be considered too. In the particular case of class imbalance measures, Batista et al. (2004) show that the harmful effects due to class imbalance are more pronounced when there is also a large overlap between the classes. Therefore, these measures should be analyzed together with measures able to capture class overlap (ex. C2 with N1). Regarding networkbased measures, the parameter in the algorithm in ECol is fixed at 0.15, although we can expect that different values may be more appropriate for distinct datasets. With the free distribution of the ECol package, interested users are able to modify this value and also other parameters (such as the distance metric employed in various measures) and test their influence in the reported results.
Finally, whist some measures are based on classification models derived from the data, others use only statistics directly derived from the data. Those that use classification models, i.e., a linear classifier or an NN classifier, are: F1v, L1, L2, L3, N3, N4. This makes these measures dependent on the classifier decisions they are based on, which in turn depends on some choices in building the classifiers, such as the algorithm to derive a linear classifier, or the distance used in nearestneighbor classification (BernadóMansilla and Ho, 2005). Other measures are based on characteristics extracted from the data only, although the N1 and the network indexes involve precomputing a distancebased graph from the dataset. Moreover, it should be noticed that all measures requiring the computation of covariances or (pseudo)inverses are time consuming, such as F1v, T3 and T4.
Smith et al. (2014a) highlight another noticeworthy issue that some measures are unable to provide an instancelevel hardness estimate. Understanding which instances are hard to classify may be valuable information, since more efforts can be devoted to them. However, many of the complexity measures originally proposed for a datasetlevel analysis can be easily adapted to give instancelevel hardness estimates. This is the case of N2, which averages the intra and interclass distances from each example to their nearest neighbors.
4 Application Areas
The data complexity measures have been applied to support of various supervised ML tasks. This section discusses some of the main applications of the complexity measures found in the relate literature. They can be roughly divided into the following categories:

data analysis, where the measures are used to understand the peculiarities of a particular dataset or domain;

data preprocessing, where the measures are employed to guide datapreprocessing tasks;

learning algorithms, where the measures are employed for understanding or in the design of ML algorithms;

metalearning, where the measures are used in the metaanalysis of classification problems, such as in choosing a particular classifier.
4.1 Data Analysis
Following the data analysis framework, some works employ the measures to better understand how the main characteristics of datasets available for learning in an application domain affect the achievable classification performance. For instance, in Lorena et al. (2012) the complexity measures are employed to analyze the characteristics of cancer gene expression data that have most impact on the predictive performance in their classification. The measures which turned out to be the most effective in such characterization were: T2 and T3 (data sparsity), C1 (class imbalance), F1 (featurebased) and N1, N2 and N3 (neighborhoodbased). The complexity measures values were also monitored after a simple feature selection strategy, which revealed the importance of such preprocessing in reducing the complexity of those highdimensional classification problems. More recently, MoránFernández et al. (2017a)
conducted a similar study with more classification and feature selection techniques and reached similar conclusions. They also tried to answer whether classification performance could be predicted by the complexity measure values in the case of the microarray datasets. In that study, the complexity measures with highlighted results were: F1 and F3 (featurebased), N1 and N2 (neighborhoodbased) when the knearest neighbor (kNN) classifier is used and L1 (linearitybased) in the case of linear classifiers.
Another interesting use of the data complexity measures has been in generating artificial datasets with controlled characteristics. This resulted in some data repositories with systematic coverage for evaluating classifiers under different challenging conditions (Macià et al., 2010; Smith et al., 2014b; Macià and BernadóMansilla, 2014; de Melo and Lorena, 2018). In (Macià et al., 2010)
a multiobjective Genetic Algorithm (GA) is employed to select subsets of instances of a dataset targetting at specific ranges of values of one or more complexity measures. In their experiments, one representative of each of the categories of
Ho and Basu (2002)’s complexity measures was chosen to be optimized: F2, N4 and T1. Later, in (Macià and BernadóMansilla, 2014) the same authors analyze the UCI repository. They experimentally observed that the majority of the UCI problems are easy to learn (only 3% were challenging for the classifiers tested). To increase the diversity of the repository, Macià and BernadóMansilla (2014) suggest to include artificial datasets carefully designed to span the complexity space, which are produced by their multiobjective GA. This gave rise to the UCI+ repository. In (de Melo and Lorena, 2018), a hillclimbing algorithm is also employed to find synthetic datasets with targetted complexity measure values. Some measures devoted to evaluate the overlapping of the classes were chosen to be optimized: F1, N1 and N3. The algorithm starts with randomly produced datasets and the labels of the examples are iteratively switched seeking to reach a given complexity measure value.4.2 Data PreProprocessing
The data complexity measures have also been used to guide data preprocessing tasks, such as Feature Selection (FS) (Liu et al., 2010), noise identification (Frenay and Verleysen, 2014) and dealing with data imbalance (He and Garcia, 2009; Fernández et al., 2018).
In FS, the measures have been used to both guide the search for the best featues in a dataset (Singh, 2003b; Okimoto et al., 2017) or to understand feature selection effects (Baumgartner et al., 2006; Pranckeviciene et al., 2006; Skrypnyk, 2011). For instance, Pranckeviciene et al. (2006) propose to quantify whether FS effectively changes the complexity of the original classification problem. They found that FS was able to increase class separability in the reduced spaces, as measured by N1, N2 and T1. Okimoto et al. (2017) assert the power of some complexity measures in ranking the features contained in synthetic datasets for which the relevant features are known a priori. As expected, featurebased measures (mainly F1) are very effective in revealing the relevant features in a dataset, although some neighborhood measures (N1 and N2) also present highlighted results. Another interesting recent work on feature selection uses a combination of the featurebased complexity measures F1, F2 and F3 to support the choice of thresholds in the number of features to be selected by FS algorithms (SeijoPardo et al., 2019).
Instance (or prototype) selection (IS) has also been the theme of various works involving the data complexity measures. In one of the first works in the area, Mollineda et al. (2005) tries to predict which instance selection algorithm should be applied to a new dataset. They report highlighted results of the F1 measure in identifying situations in which an IS technique is needed. Other works include: Leyva et al. (2014) and Cummins and Bridge (2011). Leyva et al. (2014), for instance, presents some complexity measures which are claimed to be specifically designed for characterizing IS problems. Among them is the LCS measure. Kim and Oommen (2009) perform a different analysis. They are interested in investigating whether the complexity measures can be calculated at reduced datasets while still preserving the characteristics found in the original datasets. Only separabilitymeasures are considered, among them F1, F2, F3 and N2. The results were positive for all measures, except for F1.
Under his partitioning framework, Singh (2003b)
discusses how potential outliers can be identified in a dataset. Other uses of the complexity measures in the noise identification context are:
(Smith et al., 2014a; Saéz et al., 2013; Garcia et al., 2013, 2015, 2016). Garcia et al. (2015), for example, investigate how different label noise levels affect the values of the complexity measures. Neighborhoodbased (N1, N2 and N3), featurebased (F1 and F3) and some networkbased measures (density and hubs) were found to be effective in capturing the presence of label noise in classification datasets. Two of the measures most sensitive to noise imputation were then combined to develop a new noise filter, named GraphNN.Gong and Huang (2012) found that the data complexity of a classification problem is more determinant in model performance than class imbalance, and that class imbalance amplifies the effects of data complexity. Vorraboot et al. (2012) adapted the backpropagation (BP) algorithm to take into account the class overlap and the imbalance ratio of a dataset, using the F1 featurebased measure and the imbalanceratio for binary problems. López et al. (2012) uses the F1 measure to analyze the differences between preprocessing techniques and costsensitive learning for addressing imbalanced data classification. Other works in the analysis of imbalanced classification problems include Xing et al. (2013), Anwar et al. (2014), Santos et al. (2018) and Zhang et al. (2019). More discussions on the effectiveness of data complexity analysis related to the data imbalance theme can be found in (Fernández et al., 2018).
4.3 Learning Algorithms
Data complexity measures can also be employed for analysis at the level of algorithms. These analyses can be for devising, tuning or understanding the behavior of different learning algorithms. For instance, Zhao et al. (2016) use the complexity measures to understand the data transformations performed by Extreme Learning Machines at each of their layers. They have noticed some small changes in the complexity as measured by F1, F3 and N2, which were regarded as nonsignificant.
A very popular use of the data complexity measures is to outline the domains of competence of one or more ML algorithms (Luengo and Herrera, 2015). This type of analysis allows to identify problem characteristics for which a given technique will probably succeed or fail. While improving the understanding of the capabilities and limitations of each technique, it also supports the choice of a particular technique for solving a new problem. It is possible to reformulate a learning procedure by taking into account the complexity measures too, or to devise new ML and preprocessing techniques.
In the analysis of the domains of competence of algorithms, one can cite: Ho (2000, 2002) for random decision forests; BernadóMansilla and Ho (2005) for the XCS classifier; Ho and BernadóMansilla (2006) for NN, Linear Classifier, Decision Tree, Subspace Decision Forest and Subsample Decision Forest; Flores et al. (2014) for finding datasets that fit for a
seminaive Bayesian Network Classifier
(BNC) and to recommend the best seminaive BNC to use for a new dataset; Trujillo et al. (2011) for a Genetic Programming classifier; Ciarelli et al. (2013) for incremental learning algorithms; GarciaPiquer et al. (2012); Fornells et al. (2007) for CBR; and Britto Jr et al. (2014) for the Dynamic Selection (DS) of classifiers.In Luengo and Herrera (2015) a general automatic method for extracting the domains of competence of any ML classifier is proposed. This is done by monitoring the values of the data complexity measures and relating them to the difference in the training and testing accuracies of the classifiers. Rules are extracted from the measures to identify when the classifiers will achieve a good or bad accuracy performance.
The knowledge advent from the problem complexity analysis can also be used for improving the design of existent ML techniques. For instance, Smith et al. (2014a) propose a modification of the backpropagation
algorithm for training Artificial Neural Networks (ANNs) which embed their concept of instance hardness. Therein, the error function of the BP algorithm places more emphasis on the hard instances. Other works along this line include:
Vorraboot et al. (2012) also on NN, using the measures F1 and imbalance ratio; Campos et al. (2012) in DT ensembles, using the N1 and F4 measures. Recently, Brun et al. (2018) proposed a framework for dynamic classifier selection in ensembles. It uses a subset of the complexity measures for both: selecting subsets of instances to train the pool of classifiers that compose the ensemble; and to determine the predictions that will be used for a given subproblem, which will favor classifiers trained on subproblems of similar complexity to the query subproblem. They have selected one measure from each of the Ho and Basu (2002)’s original categories which showed low Pearson correlation with each other to be optimized by a GA suited for DS: F1, N2 and N4.On the other hand, some works have devised new approaches for data classification based on the information of the complexity measures. This is the case of Lorena and de Carvalho (2010), in which the measures F1 and F2 are used as splitting criteria for decomposing multiclass problems. Quiterio and Lorena (2018) also work on the decomposition of multiclass problems, using the complexity measures to place the binary classifiers in Directed Acyclic Graph structures. No specific complexity measure among those tested in the paper (namely F1, F3, N1, N2 and T1) could be regarded as best suited for optimizing the DAG structures, although all of them were suitable choices for evaluating the binary classifiers. Sun et al. (2019) perform hierarchical partitions of the classes minimizing classification complexity, which are estimated according to the measures F1, F2, F3, N2, N3 and a new measure based on centroids introduced in their work. The best experimental results were obtained for the measures F1, F3 and centroidbased.
Another task that can be supported by the estimates on problem complexity is to tune the parameters of the ML techniques for a given problem. In He et al. (2015) the data complexity measures are applied to describe the leak quantification problem. They employ one representative measure of each of the Ho and Basu (2002)’s categories: F2, N1 and T1. In addition, a parameter tuning procedure which minimizes data complexity under some domainspecific constraints is proposed. Measures N1 and T1 achieved better results. Nojima et al. (2011) use the complexity measures to specify the parameter values of fuzzy classifiers. Some decision rules for binary classification problems based on measures F4, L1, L2, N1 and T2 are reported. N4 is also mentioned as key measure in the case of multiclass problems.
4.4 MetaLearning
In Metalearning (MtL), metaknowledge about the solutions of previous problems is used to aid the solution of a new problem (Vilalta and Drissi, 2002). For this, a metadataset composed of datasets for which the solutions are known is usually built. They must be described by metafeatures, which is how the complexity measures are mainly used in this area. Some works previously described have made use of metalearning so they also fall in this category (e.g., Smith et al. (2014b); Leyva et al. (2014); Nojima et al. (2011); Zhang et al. (2019)).
The work of Mollineda et al. (2006) is one of the first to present a general metalearning framework based on a number of data complexity measures. Walt and Barnard (2007) employ the data complexity measures to characterize classification problems in a metalearning setup designed to predict the expected accuracy of some ML techniques. Krijthe et al. (2012) compare classifier selection using crossvalidation with metalearning. Ren and Vale (2012) use the data complexity measures F1, F2, F3, N1, N2, T1 and T2 to predict the behavior of the NN classifier. In Garcia et al. (2016) an MtL recommender system able to predict the expected performance of noise filters in noisy data identification tasks is presented. For such, a metalearning database is created, containing metafeatures, characteristics extracted from several corrupted datasets, along with the performance of some noise filters when applied to these datasets. Along with some standard metalearning metafeatures, the complexity measures N1 and N3 have a higher contribution to the prediction results. More recent works on metalearning include: Cruz et al. (2015); das Dôres et al. (2016); Roy et al. (2016); Parmezan et al. (2017); Cruz et al. (2017a, 2018); Garcia et al. (2018); Shah et al. (2018). In (Garcia et al., 2018)
, for example, all of the complexity measures described in this work were employed to generate regression models able to predict the accuracies of four classifiers with very distinct biases: ANN, decision tree, kNN and SVM. The estimated models were effective in such predictions. The topranked metafeatures chosen by one particular regression technique (Random Forest  RF) were N3, N1, N2, density and T1. All of them regard neighborhoodbased information from the data (in the case of density, in the form of a graph built from the data).
Another interesting usage of the complexity measures in the metaanalysis of classification problems is presented in (Muñoz et al., 2018)
. There an instance space is built based on metafeatures extracted from a large set of classification problems, along with the performance of multiple classifiers. Among the metafeatures used are the complexity measures F3, F4, L2, N1 and N4. The instance space framework provides an interesting overview of which are the hardest and easiest datasets and also to identify strengths and weaknesses of individual classifiers. The paper also presents a method to generate new datasets which better span the instance space.
4.5 Summary
A summary of the main applications of the data complexity measures found in the literature is presented in Table 2. It can be observed that these measures have been mainly employed in the characterization of the domains of competence of various learning and also preprocessing techniques, by revealing when they will perform well or not. These are generalized to the use of the measures as metafeatures for describing datasets in metalearning studies.
Category  Subtype  References 

Data Analysis  Domain understanding  Lorena et al. (2012); Kamath et al. (2008); GarcíaCallejas and Araújo (2016) 
MoránFernández et al. (2017a)  
Data generation  Macià and BernadóMansilla (2014); Macia et al. (2008); Macià et al. (2010, 2013)  
(Smith et al., 2014b; de Melo and Lorena, 2018; Muñoz et al., 2018)  
Data Preprocessing  Feature Selection  Singh (2003b); Okimoto et al. (2017); Baumgartner et al. (2006); Pranckeviciene et al. (2006) 
Skrypnyk (2011); SeijoPardo et al. (2019)  
Instance Selection  Mollineda et al. (2005); Leyva et al. (2014); Cummins and Bridge (2011); Kim and Oommen (2009)  
Noise identification  Singh (2003b); Smith et al. (2014a); Saéz et al. (2013); Garcia et al. (2013, 2015, 2016)  
Class imbalance  Gong and Huang (2012); Vorraboot et al. (2012); López et al. (2012); Xing et al. (2013)  
Anwar et al. (2014); Zhang et al. (2019); Santos et al. (2018)  
Learning algorithms  Domain of competence  BernadóMansilla and Ho (2005); Ho and BernadóMansilla (2006); Flores et al. (2014) 
Trujillo et al. (2011); Ciarelli et al. (2013); GarciaPiquer et al. (2012); Fornells et al. (2007)  
Ho (2000); Britto Jr et al. (2014); Luengo and Herrera (2015); Lucca et al. (2017)  
Algorithm design  Smith et al. (2014a); Vorraboot et al. (2012); Campos et al. (2012); Brun et al. (2018)  
Algorithm understanding  Zhao et al. (2016)  
Multiclass decomposition  Lorena and de Carvalho (2010); Quiterio and Lorena (2018); MoránFernández et al. (2017b)  
Sun et al. (2019)  
Parameter tuning  He et al. (2015); Nojima et al. (2011)  
Metalearning  Metafeatures  Smith et al. (2014b); Leyva et al. (2014); Garcia et al. (2016, 2018); Nojima et al. (2011) 
Mollineda et al. (2006); Walt and Barnard (2007); Krijthe et al. (2012); Ren and Vale (2012)  
das Dôres et al. (2016); Roy et al. (2016); Parmezan et al. (2017); Cruz et al. (2015, 2017a)  
Muñoz et al. (2018); Zhang et al. (2019); Shah et al. (2018) 
Concerning the usage of the individual measures, we can notice a variation per domain. As expected, featurebased measures are quite effective in FS. Among them, F1 is the most used and has shown highlighted results also in instance selection and in class imbalance analysis. LSC was proposed in the IS context. Neighborhoodbased measures (mainly N1, N2 and N3) also show detached results in different domains, such as FS, noise identification and metalearning. But one should be aware that in most of the reviewed work there was no clear evaluation on the contribution of each of the complexity measures values in the results achieved. Indeed, most of the related work perform an adhoc selection of which complexity measures are to be used (for example, one representative measure per category). Since each measure provides a distinct perspective on classification complexity, a combination of different measures is advised. Nonetheless, whether there is a subset of the complexity measures that can be considered core to stress the difficulty of problems from different application domains is still an openissue.
5 Conclusion
This paper reviewed the main data complexity measures from the literature. These indices allow to characterize the difficulty of a classification problem from the perspectives of data geometry and distribution within or across the classes. They were first proposed and analyzed in Ho and Basu (2002) and have since been extensively used in the analysis and development of classification and preprocessing techniques.
The original complexity measures and other measures found in related literature were briefly presented. Despite the presence of many methods for measuring the complexity of classification problems, they often share similar concepts. There has not been a study comparing them to reveal which ones can extract more distinct aspects regarding data complexity. Besides the characteristics of each individual measure highlighted alongside their definitions, we present next some general discussions about each category of complexity measures.
In the case of the featurebased complexity measures, there is an expectation that each feature has a certain contribution to the discrimination task, and that the axis representing the feature can be interpreted as it is. This is more likely to be true for problems where the features are meaningful explanatory variables each contributing somewhat independently to the classification. It is particularly less likely to be true in classification problems where sensory signals are directly taken as input, such as pixel values in images, where a natural unit of discriminatory information tends to involve a larger group of features (such as a patch of colors displayed over multiple pixels). For those cases, transformation of the raw feature values, such as by a directional vector projection, becomes essential. The second issue is that as we examine the overlap of the feature value ranges, there is an expectation that the unseen values in an interval that spans the seen values contribute to the discrimination task in a similar way as the seen values, i.e., there is continuity in the class definition w.r.t. that feature. This tends to be true for features in a continuous numerical scale, and is less likely for other cases. For categorical features, the notion of value ranges degenerates into specific values and several measures in this family have difficulties.
The measures in the linearity family focus on the perspective of linear separability, which has a long history of being used as a characterization of classification difficulty. It was involved in the early debates of the limits of certain classifier’s capabilities (e.g. the debate on the perceptron in Minsky and Papert (1969)). One issue of concern is that linear separability is often characteristic of sparse data sets – consider the extreme case where only one training point is available from each class in an arbitrary classification problem, and in that case linear separability of the training data does not give much information about the nature of the underlying task. Sparse datasets in high dimensional space are also likely to be linearly separable (see, for example, (Costa et al., 2009)), which motivates techniques like SVMs that use a feature transformation to map the data to a high dimensional space where simple linear classifiers suffice. The interactive effects of this type of measures with data size, data density, and dimensionality are illustrative of the challenges involved in data complexity discussions. Therefore the complexity evaluations need to be anchored first on fixed datasets, and followed by discussions of changes in responses to the other influences.
Measures in this neighborhoodbased family characterize the datasets in ways different from those of the featurebased family and the linearitybased family. They use a distance function to summarize the relationship between points. This is best fitted for datasets where the features are on a comparable scale (e.g. perpixel intensity values) such that a natural metric exists. For datasets that involve features of heterogeneous types and scales, a scalenormalization step or a suitable weighting scheme is needed for a summarizing metric to be properly defined. The usefulness of the measures depends critically on whether such a metric can be obtained. The Gower distance metric employed in ECol is a simple alternative for dealing with features of different types and scales, but more sophisticate distance functions could be used instead (Wilson and Martinez, 1997). In addition, since these measures are influenced by withinclass data distributions as well as by the data distributions near the class boundaries, the information they convey may include more than what is relevant to the discrimination task, which may cause drownout of the critical signal about classification complexity.
The network based measures regard on the structure of the data in the input space. They may complement the previous measures presented, although they also consider the neighborhood of examples for obtaining the graph representation. It should be noticed that a number of other complex network measures can be extracted from the graph built, as well as other strategies can be used to obtain the graph representation. The strategy chosen to built the graph from a learning dataset considers both the proximity of the examples (NN) and the data label information (pruning step). Herewith, we expect to get an overview of both intra and interclass relationships.
All measures from the dimensionality group rely only on the numbers of examples and features in a dataset, disregarding the label information. Therefore, they do not give any indicative of boundary complexity, but rather give a very simplified and naïve overview on data sparsity. As discussed in the paper introduction, data sparsity is one of the factors that may affect the complexity of a classification problem. Indeed, datasets with a high dimensionality and a low number of examples tend to be distributed sparsely. In many cases this can make the classification problem look simpler than it really is so that simple classification models may not generalize well to new data points that occupy regions formerly underrepresented in the training dataset.
The measures of the class imbalance category regard on the number of examples per class. As in the case of the dimensionality measures, they do not allow to directly estimate the complexity of the classification boundary. Rather, they regard on another aspect which may influence the performance of many ML classification techniques, which is the underrepresentation of one or more classes in relation to others.
This work also provides an R package with an implementation of a set of 22 complexity measures from the previous categories. The package is expected to give interested researchers a quick start into the field. An immediate line of followup work is to evaluate these measures empirically and try to: (i) identify those measures with most distinct concepts, since many of them have similar computation; and (ii) compare their ability in revealing the complexity of a diverse set of classification problems. This type of investigation is expected to yield a reduced subset of core measures able to capture the most critical aspects of classification complexity.
Lastly, the main use cases where the measures have been applied were presented. The most common use of the measures is to characterize datasets in metalearning studies or the domain of competence of learning and preprocessing techniques. Nonetheless, more contributions remain possible in employing the conclusions of these studies to adapt and propose new learning and preprocessing techniques. For instance, relatively few works have been done in devising new learning schemes and preprocessing techniques based on the complexity measures. This points to the potentials of these measures that remain poorly explored. We believe that a better understanding of the characteristics of a given problem shall be the key to support the design of techniques with better predictive results.
Another direction that awaits to be better explored is how one can use the complexity measures to evaluate different formulations of a problem, in terms of how classes are defined or chosen, in domains where there is flexibility in such choices. An example is a text categorization task, where one may have some limited freedom to choose what is to be considered a category. Better choices can lead to lower error rates even if the classifier technology stays the same. Here the data complexity measures can serve as figures of merit to evaluate alternative class definitions.
Acknowledgements
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior  Brasil (CAPES)  Finance Code 001. The authors would also like to thank the financial support of the foundations FAPESP (grant 2012/226088), CNPq (grants 308858/20140 and 305291/20173), CAPES (grant 88887.162551/201800) and CAPESCOFECUB.
References
 Ali and Smith [2006] Shawkat Ali and Kate A Smith. On learning algorithm selection for classification. Applied Soft Computing, 6(2):119–138, 2006.
 Anwar et al. [2014] Nafees Anwar, Geoff Jones, and Siva Ganesh. Measurement of data complexity for classification problems with unbalanced data. Statistical Analysis and Data Mining, 7(3):194–211, 2014.
 Armano [2015] Giuliano Armano. A direct measure of discriminant and characteristic capability for classifier building and assessment. Information Sciences, 325:466–483, 2015.
 Armano and Tamponi [2016] Giuliano Armano and Emanuele Tamponi. Experimenting multiresolution analysis for identifying regions of different classification complexity. Pattern Analysis and Applications, 19(1):129–137, 2016.
 Basu and Ho [2006] Mitra Basu and Tin K Ho. Data complexity in pattern recognition. Springer, 2006.
 Batista et al. [2004] Gustavo E A P A Batista, Ronaldo C Prati, and Maria C Monard. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explorations Newsletter, 6(1):20–29, 2004.
 Baumgartner et al. [2006] Richard Baumgartner, Tin K Ho, Ray Somorjai, Uwe Himmelreich, and Tania Sorrell. Complexity of magnetic resonance spectrum classification. In Data Complexity in Pattern Recognition, pages 241–248, 2006.

BernadóMansilla and Ho [2005]
Ester BernadóMansilla and Tin K Ho.
Domain of competence of xcs classifier system in complexity
measurement space.
IEEE Transactions on Evolutionary Computation
, 9(1):82–104, 2005.  Bottou and Lin [2007] Léon Bottou and ChihJen Lin. Support vector machine solvers. Large scale kernel machines, 3(1):301–320, 2007.
 Britto Jr et al. [2014] Alceu S Britto Jr, Robert Sabourin, and Luiz E S Oliveira. Dynamic selection of classifiers  a comprehensive review. Pattern Recognition, 47(11):3665–3680, 2014.
 Brun et al. [2018] André L Brun, Alceu S Britto Jr, Luiz S Oliveira, Fabricio Enembreck, and Robert Sabourin. A framework for dynamic classifier selection oriented by the classification problem difficulty. Pattern Recognition, 76:175–190, 2018.

Caliński and Harabasz [1974]
Tadeusz Caliński and Jerzy Harabasz.
A dendrite method for cluster analysis.
Communications in Statisticstheory and Methods, 3(1):1–27, 1974. 
Campos et al. [2012]
Yoisel Campos, Carlos Morell, and Francesc J Ferri.
A local complexity based combination method for decision forests trained with highdimensional data.
In 12th International Conference on Intelligent Systems Design and Applications (ISDA), pages 194–199, 2012. 
Charte et al. [2016]
Francisco Charte, Antonio Rivera, María J del Jesus, and Francisco
Herrera.
On the impact of dataset complexity and sampling strategy in
multilabel classifiers performance.
In
11th International Conference on Hybrid Artificial Intelligence Systems (HAIS)
, pages 500–511, 2016.  Ciarelli et al. [2013] Patrick M Ciarelli, Elias Oliveira, and Evandro O T Salles. Impact of the characteristics of data sets on incremental learning. Artificial Intelligence Research, 2(4):63–74, 2013.
 Costa et al. [2009] Ivan G Costa, Ana C Lorena, Liciana R M P y Peres, and Marcilio C P de Souto. Using supervised complexity measures in the analysis of cancer gene expression data sets. In Brazilian Symposium on Bioinformatics, pages 48–59, 2009.
 Cristianini and ShaweTaylor [2000] Nello Cristianini and John ShaweTaylor. An introduction to support vector machines and other kernelbased learning methods. Cambridge university press, 2000.
 Cruz et al. [2015] Rafael M O Cruz, Robert Sabourin, George D C Cavalcanti, and Tsang Ing Ren. METADES: A dynamic ensemble selection framework using metalearning. Pattern recognition, 48(5):1925–1935, 2015.
 Cruz et al. [2017a] Rafael M O Cruz, Robert Sabourin, and George D C Cavalcanti. METADES.Oracle: metalearning and feature selection for dynamic ensemble selection. Information fusion, 38:84–103, 2017a.
 Cruz et al. [2017b] Rafael M O Cruz, Hiba H Zakane, Robert Sabourin, and George D C Cavalcanti. Dynamic ensemble selection vs knn: why and when dynamic selection obtains higher classification performance? In 17th International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1–6, 2017b.
 Cruz et al. [2018] Rafael M O Cruz, Robert Sabourin, and George D C Cavalcanti. Dynamic classifier selection: Recent advances and perspectives. Information Fusion, 41:195–216, 2018.
 Cummins [2013] Lisa Cummins. Combining and Choosing Case Base Maintenance Algorithms. PhD thesis, National University of Ireland, Cork, 2013.
 Cummins and Bridge [2011] Lisa Cummins and Derek Bridge. On dataset complexity for case base maintenance. In 19th International Conference on CaseBased Reasoning (ICCBR), pages 47–61. 2011.
 das Dôres et al. [2016] Silvia N das Dôres, Luciano Alves, Duncan D Ruiz, and Rodrigo C Barros. A metalearning framework for algorithm recommendation in software fault prediction. In 31st Annual ACM Symposium on Applied Computing (SAC), pages 1486–1491, 2016.
 de Melo and Lorena [2018] Vinícius V de Melo and Ana C Lorena. Using complexity measures to evolve synthetic classification datasets. In International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2018.
 Dong and Kothari [2003] Ming Dong and Rishabh P Kothari. Feature subset selection using a new definition of classificability. Pattern Recognition Letters, 24:1215–1225, 2003.
 Elizondo et al. [2012] David A Elizondo, Ralph Birkenhead, Matias Gamez, Noelia Garcia, and Esteban Alfaro. Linear separability and classification complexity. Expert Systems with Applications, 39(9):7796–7807, 2012.
 Fernández et al. [2018] Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, Bartosz Krawczyk, and Francisco Herrera. Learning from imbalanced data sets. Springer, 2018.
 Flores et al. [2014] María J Flores, José A Gámez, and Ana M Martínez. Domains of competence of the seminaive bayesian network classifiers. Information Sciences, 260:120–148, 2014.
 Fornells et al. [2007] Albert Fornells, Elisabet Golobardes, Josep M Martorell, Josep M Garrell, Núria Macià, and Ester Bernadó. A methodology for analyzing case retrieval from a clustered case memory. In 7th International Conference on CaseBased Reasoning (ICCBR), pages 122–136. 2007.
 Frenay and Verleysen [2014] Benoit Frenay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845 – 869, 2014.
 Garcia et al. [2013] Luís P F Garcia, André C P L F de Carvalho, and Ana C Lorena. Noisy data set identification. In 8th International Conference on Hybrid Artificial Intelligent Systems (HAIS), pages 629–638. 2013.
 Garcia et al. [2015] Luís P F Garcia, André C P L F de Carvalho, and Ana C Lorena. Effect of label noise in the complexity of classification problems. Neurocomputing, 160:108–119, 2015.
 Garcia et al. [2016] Luís P F Garcia, André C P L F de Carvalho, and Ana C Lorena. Noise detection in the metalearning level. Neurocomputing, 176:14–25, 2016.
 Garcia et al. [2018] Luís P F Garcia, Ana C Lorena, Marcilio C P de Souto, and Tin Kam Ho. Classifier recommendation using data complexity measures. In 24th International Conference on Pattern Recognition (ICPR), pages 874–879, 2018.
 GarcíaCallejas and Araújo [2016] David GarcíaCallejas and Miguel B Araújo. The effects of model and data complexity on predictions from species distributions models. Ecological Modelling, 326:4–12, 2016.
 GarciaPiquer et al. [2012] Alvaro GarciaPiquer, Albert Fornells, Albert OrriolsPuig, Guiomar Corral, and Elisabet Golobardes. Data classification through an evolutionary approach based on multiple criteria. Knowledge and information Systems, 33(1):35–56, 2012.
 Gong and Huang [2012] Rongsheng Gong and Samuel H Huang. A kolmogorov–smirnov statistic based segmentation approach to learning from imbalanced datasets: With application in property refinance prediction. Expert Systems with Applications, 39(6):6192–6200, 2012.
 Gower [1971] John Gower. A general coefficient of similarity and some of its properties. Biometrics, 27(4):857–871, 1971.
 He and Garcia [2009] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284, 2009.
 He et al. [2015] ZhiMin He, Patrick P K Chan, Daniel S Yeung, Witold Pedrycz, and Wing W Y Ng. Quantification of sidechannel information leaks based on data complexity measures for web browsing. International Journal of Machine Learning and Cybernetics, 6(4):607–619, 2015.
 Ho [2000] Tin K Ho. Complexity of classification problems and comparative advantages of combined classifiers. In Multiple Classifier Systems (MCS), pages 97–106. 2000.
 Ho [2002] Tin K Ho. A data complexity analysis of comparative advantages of decision forest constructors. Pattern Analysis and Applications, (5):102–112, 2002.
 Ho and Basu [2002] Tin K Ho and Mitra Basu. Complexity measures of supervised classification problems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(3):289–300, 2002.
 Ho and BernadóMansilla [2006] Tin K Ho and Ester BernadóMansilla. Classifier domains of competence in data complexity space. In Data complexity in pattern recognition, pages 135–152, 2006.
 Ho et al. [2006] Tin K Ho, Mitra Basu, and Martin H C Law. Measures of geometrical complexity in classification problems. In Data Complexity in Pattern Recognition, pages 1–23, 2006.
 Hoekstra and Duin [1996] Aarnoud Hoekstra and Robert P W Duin. On the nonlinearity of pattern classifiers. In 13th International Conference on Pattern Recognition (ICPR), volume 4, pages 271–275, 1996.
 Hu et al. [2010] Qinghua Hu, Witold Pedrycz, Daren Yu, and Jun Lang. Selecting discrete and continuous features based on neighborhood decision error minimization. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 40(1):137–150, 2010.
 Kamath et al. [2008] Vidya Kamath, Timothy J Yeatman, and Steven A Eschrich. Toward a measure of classification complexity in gene expression signatures. In 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), pages 5704–5707, 2008.
 Kim and Oommen [2009] SangWoon Kim and John Oommen. On using prototype reduction schemes to enhance the computation of volumebased interclass overlap measures. Pattern Recognition, 42(11):2695–2704, 2009.
 Kolaczyk [2009] Eric D Kolaczyk. Statistical Analysis of Network Data: Methods and Models. Springer Series in Statistics. Springer, 2009.
 Kotsiantis and Kanellopoulos [2006] Sotiris Kotsiantis and Dimitris Kanellopoulos. Discretization techniques: A recent survey. GESTS International Transactions on Computer Science and Engineering, 32(1):47–58, 2006.
 Krijthe et al. [2012] Jesse H Krijthe, Tin K Ho, and Marco Loog. Improving crossvalidation based classifier selection using metalearning. In 21st International Conference on Pattern Recognition (ICPR), pages 2873–2876, 2012.

Lebourgeois and Emptoz [1996]
Frank Lebourgeois and Hubert Emptoz.
Pretopological approach for supervised learning.
In 13th International Conference on Pattern Recognition, volume 4, pages 256–260, 1996.  Leyva et al. [2014] Enrique Leyva, Antonio González, and Raúl Pérez. A set of complexity measures designed for applying metalearning to instance selection. IEEE Transactions on Knowledge and Data Engineering, 27(2):354–367, 2014.
 Ling and AbuMostafa [2006] Li Ling and Yaser S AbuMostafa. Data complexity in machine learning. Technical Report CaltechCSTR:2006.004, California Institute of Technology, 2006.
 Liu et al. [2010] Huan Liu, Hiroshi Motoda, Rudy Setiono, and Zheng Zhao. Feature selection: An ever evolving frontier in data mining. In 4th International Workshop on Feature Selection in Data Mining (FSDM), volume 10, pages 4–13, 2010.
 López et al. [2012] Victoria López, Alberto Fernández, Jose G MorenoTorres, and Francisco Herrera. Analysis of preprocessing vs. costsensitive learning for imbalanced classification. open problems on intrinsic data characteristics. Expert Systems with Applications, 39(7):6585–6608, 2012.
 Lorena and de Carvalho [2010] Ana C Lorena and André C P L F de Carvalho. Building binarytreebased multiclass classifiers using separability measures. Neurocomputing, 73(1618):2837–2845, 2010.
 Lorena et al. [2008] Ana C Lorena, André C P L F de Carvalho, and João M P Gama. A review on the combination of binary classifiers in multiclass problems. Artificial Intelligence Review, 30(1):19–37, 2008.
 Lorena et al. [2012] Ana C Lorena, Ivan G Costa, Newton Spolaôr, and Marcilio C P Souto. Analysis of complexity indices for classification problems: Cancer gene expression data. Neurocomputing, 75(1):33–42, 2012.
 Lorena et al. [2018] Ana C Lorena, Aron I Maciel, Pericles B C Miranda, Ivan G Costa, and Ricardo B C Prudêncio. Data complexity metafeatures for regression problems. Machine Learning (accepted), 2018.
 Lucca et al. [2017] Giancarlo Lucca, Jose Sanz, Graçaliz P Dimuro, Benjamín Bedregal, and Humberto Bustince. Analyzing the behavior of aggregation and preaggregation functions in fuzzy rulebased classification systems with data complexity measures. In 10th Conference of the European Society for Fuzzy Logic and Technology (IWIFSGN), pages 443–455, 2017.
 Luengo and Herrera [2015] Julián Luengo and Francisco Herrera. An automatic extraction method of the domains of competence for learning classifiers using data complexity measures. Knowledge and Information Systems, 42(1):147–180, 2015.
 Macià [2011] Núria Macià. Data Complexity in Supervised Learning: a farreaching implication. PhD thesis, La Salle, Universitat Ramon Llull, 2011.
 Macià and BernadóMansilla [2014] Núria Macià and Ester BernadóMansilla. Towards uci+: A mindful repository design. Information Sciences, 261:237–262, 2014.
 Macia et al. [2008] Núria Macia, Ester BernadóMansilla, and Albert OrriolsPuig. Preliminary approach on synthetic data sets generation based on class separability measure. In 19th International Conference on Pattern Recognition (ICPR), pages 1–4, 2008.
 Macià et al. [2010] Núria Macià, Albert OrriolsPuig, and Ester BernadóMansilla. In search of targetedcomplexity problems. In 12th annual conference on Genetic and evolutionary computation, pages 1055–1062, 2010.
 Macià et al. [2013] Núria Macià, Ester BernadóMansilla, Albert OrriolsPuig, and Tin Kam Ho. Learner excellence biased by data set selection: A case for data characterisation and artificial data sets. Pattern Recognition, 46(3):1054–1066, 2013.
 Malina [2001] Witold Malina. Twoparameter fisher criterion. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 31(4):629–636, 2001.
 Ming and Vitanyi [1993] Li Ming and Paul Vitanyi. An Introduction to Kolmogorov Complexity and Its Applications. Springer, 1993.
 Minsky and Papert [1969] Marvin Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. The MIT Press, Cambridge MA, 1969.
 Mollineda et al. [2005] Ramón A Mollineda, José S Sánchez, and José M Sotoca. Data characterization for effective prototype selection. In 2nd Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), pages 27–34, 2005.
 Mollineda et al. [2006] Ramón A Mollineda, José S Sánchez, and José M Sotoca. A metalearning framework for pattern classification by means of data complexity measures. Inteligencia Artificial, 10(29):31–38, 2006.
 Morais and Prati [2013] Gleison Morais and Ronaldo C Prati. Complex network measures for data set characterization. In 2nd Brazilian Conference on Intelligent Systems (BRACIS), pages 12–18, 2013.
 MoránFernández et al. [2017a] Laura MoránFernández, Verónica BolónCanedo, and Amparo AlonsoBetanzos. Can classification performance be predicted by complexity measures? a study using microarray data. Knowledge and Information Systems, 51(3):1067–1090, 2017a.
 MoránFernández et al. [2017b] Laura MoránFernández, Verónica BolónCanedo, and Amparo AlonsoBetanzos. On the use of different base classifiers in multiclass problems. Progress in Artificial Intelligence, 6(4):315–323, 2017b.
 Mthembu and Marwala [2008] Linda Mthembu and Tshilidzi Marwala. A note on the separability index. arXiv preprint arXiv:0812.1107, 2008.
 Muñoz et al. [2018] Mario A Muñoz, Laura Villanova, Davaatseren Baatar, and Kate SmithMiles. Instance spaces for machine learning classification. Machine Learning, 107(1):109–147, 2018.
 Nojima et al. [2011] Yusuke Nojima, Shinya Nishikawa, and Hisao Ishibuchi. A metafuzzy classifier for specifying appropriate fuzzy partitions by genetic fuzzy rule selection with data complexity measures. In IEEE International Conference on Fuzzy Systems (FUZZ), pages 264–271, 2011.
 Okimoto et al. [2017] Lucas Chesini Okimoto, Ricardo Manhães Savii, and Ana Carolina Lorena. Complexity measures effectiveness in feature selection. In 6th Brazilian Conference on Intelligent Systems (BRACIS), pages 91–96, 2017.
 OrriolsPuig et al. [2010] Albert OrriolsPuig, Núria Macià, and Tin K Ho. Documentation for the data complexity library in c++. Technical report, La Salle  Universitat Ramon Llull, 2010.
 Parmezan et al. [2017] Antonio R S Parmezan, Huei D Lee, and Feng C Wu. Metalearning for choosing feature selection algorithms in data mining: Proposal of a new framework. Expert Systems with Applications, 75:1–24, 2017.
 Pranckeviciene et al. [2006] Erinija Pranckeviciene, Tin K Ho, and Ray Somorjai. Class separability in spaces reduced by feature selection. In 18th International Conference on Pattern Recognition (ICPR), volume 2, pages 254–257, 2006.
 Quiterio and Lorena [2018] Thaise M Quiterio and Ana C Lorena. Using complexity measures to determine the structure of directed acyclic graphs in multiclass classification. Applied Soft Computing, 65:428–442, 2018.
 Ren and Vale [2012] George D C Cavalcantiand Tsang I Ren and Breno A Vale. Data complexity measures and nearest neighbor classifiers: A practical analysis for metalearning. In 24th International Conference on Tools with Artificial Intelligence (ICTAI), volume 1, pages 1065–1069, 2012.
 Roy et al. [2016] Anandarup Roy, Rafael M O Cruz, Robert Sabourin, and George D C Cavalcanti. Metalearning recommendation of default size of classifier pool for METADES. Neurocomputing, 216:351–362, 2016.
 Saéz et al. [2013] José A Saéz, Julián Luengo, and Francisco Herrera. Predicting noise filtering efficacy with data complexity measures for nearest neighbor classification. Pattern Recognition, 46(1):355–364, 2013.
 Santos et al. [2018] Miriam Seoane Santos, Jastin Pompeu Soares, Pedro Henrigues Abreu, Helder Araujo, and Joao Santos. Crossvalidation for imbalanced datasets: Avoiding overoptimistic and overfitting approaches. IEEE Computational Intelligence Magazine, 13(4):59–76, 2018.
 SeijoPardo et al. [2019] Borja SeijoPardo, Verónica BolónCanedo, and Amparo AlonsoBetanzos. On developing an automatic threshold applied to feature selection ensembles. Information Fusion, 45:227–245, 2019.
 Shah et al. [2018] Rushit Shah, Varun Khemani, Michael Azarian, Michael Pecht, and Yan Su. Analyzing data complexity using metafeatures for classification algorithm selection. In Prognostics and System Health Management Conference (PHMChongqing), pages 1280–1284, 2018.
 Singh [2003a] Sameer Singh. Multiresolution estimates of classification complexity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12):1534–1539, 2003a.
 Singh [2003b] Sameer Singh. PRISM: a novel framework for pattern recognition. Pattern Analysis and Applications, 6(2):134–149, 2003b.
 Skrypnyk [2011] Iryna Skrypnyk. Irrelevant features, class separability, and complexity of classification problems. In 23rd IEEE International Conference on Tools with Artificial Intelligence (ICTAI), pages 998–1003, 2011.
 Smith [1968] Fred W Smith. Pattern classifier design by linear programming. IEEE Transactions on Computers, C17(4):367–372, 1968.
 Smith et al. [2014a] Michael R Smith, Tony Martinez, and Christophe GiraudCarrier. An instance level analysis of data complexity. Machine Learning, 95(2):225–256, 2014a.
 Smith et al. [2014b] Michael R Smith, Andrew White, Christophe GiraudCarrier, and Tony Martinez. An easy to use repository for comparing and improving machine learning algorithm usage. arXiv preprint arXiv:1405.7292, 2014b.
 SmithMiles [2009] Kate A SmithMiles. Crossdisciplinary perspectives on metalearning for algorithm selection. ACM Computing Surveys (CSUR), 41(1):1–26, 2009.
 Sotoca et al. [2005] José M Sotoca, José Sánchez, and Ramón A Mollineda. A review of data complexity measures and their applicability to pattern classification problems. In Actas del III Taller Nacional de Minería de Dados y Aprendizaje (TAMIDA), pages 77–83, 2005.
 Souto et al. [2010] Marcilio C P Souto, Ana C Lorena, Newton Spolaôr, and Ivan G Costa. Complexity measures of supervised classification tasks: a case study for cancer gene expression data. In International Joint Conference on Neural Networks (IJCNN), pages 1352–1358, 2010.
 Sun et al. [2019] MengXin Sun, KunHong Liu, QingQiang Wu, QingQi Hong, BeiZhan Wang, and Haiying Zhang. A novel ECOC algorithm for multiclass microarray data classification based on data complexity analysis. Pattern Recognition, 90:346–362, 2019.

Tanwani and Farooq [2010]
Ajay K Tanwani and Muddassar Farooq.
Classification potential vs. classification accuracy: a comprehensive study of evolutionary algorithms with biomedical datasets.
Learning Classifier Systems, 6471:127–144, 2010.  Trujillo et al. [2011] Leonardo Trujillo, Yuliana Martínez, Edgar GalvánLópez, and Pierrick Legrand. Predicting problem difficulty for genetic programming applied to data classification. In 13th annual conference on Genetic and evolutionary computation (GECCO), pages 1355–1362, 2011.
 Vilalta and Drissi [2002] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of metalearning. Artificial Intelligence Review, 18(2):77–95, 2002.
 Vorraboot et al. [2012] Piyanoot Vorraboot, Suwanna Rasmequan, Chidchanok Lursinsap, and Krisana Chinnasarn. A modified error function for imbalanced dataset classification problem. In 7th International Conference on Computing and Convergence Technology (ICCCT), pages 854–859, 2012.
 Walt and Barnard [2007] Christiaan V D Walt and Etienne Barnard. Measures for the characterisation of patternrecognition data sets. In 18th Annual Symposium of the Pattern Recognition Association of South Africa (PRASA), 2007.
 Wilson and Martinez [1997] D Randall Wilson and Tony R Martinez. Improved heterogeneous distance functions. Journal of artificial intelligence research, 6:1–34, 1997.
 Wolpert [1996] David H Wolpert. The lack of a priori distinctions between learning algorithms. Neural computation, 8(7):1341–1390, 1996.
 Xing et al. [2013] Yan Xing, Hao Cai, Yanguang Cai, Ole Hejlesen, and Egon Toft. Preliminary evaluation of classification complexity measures on imbalanced data. In Chinese Intelligent Automation Conference: Intelligent Information Processing", pages 189–196, 2013.
 Zhang et al. [2019] Xueying Zhang, Ruixian Li, Bo Zhang, Yunxiang Yang, Jing Guo, and Xiang Ji. An instancebased learning recommendation algorithm of imbalance handling methods. Applied Mathematics and Computation, 351:204 – 218, 2019.
 Zhao et al. [2016] Xingmin Zhao, Weipeng Cao, Hongyu Zhu, Zhong Ming, and Rana Aamir Raza Ashfaq. An initial study on the rank of input matrix for extreme learning machine. International Journal of Machine Learning and Cybernetics, pages 1–13, 2016.
 Zhu et al. [2005] Xiaojin Zhu, John Lafferty, and Ronald Rosenfeld. Semisupervised learning with graphs. PhD thesis, Carnegie Mellon University, Language Technologies Institute, School of Computer Science, 2005.
 Zubek and Plewczynski [2016] Julian Zubek and Dariusz M Plewczynski. Complexity curve: a graphical measure of data complexity and classifier performance. PeerJ Computer Science, 2:e76, 2016.