This paper extends this line of research by investigating a different implementation of Isolation Similarity. We provide a formal proof of the characteristic of the Isolation Similarity for the first time since its introduction. In addition, we focus on using Isolation Similarity to improve the clustering performance of density-based clustering.
This paper identifies shortcomings of tree-based method currently employed in inducing Isolation Similarities [Ting, Zhu, and Zhou2018]. Instead, we investigate a different method to induce the data dependent Isolation Similarity, and evaluate its clustering performance using DBSCAN [Ester et al.1996] in comparison with the two existing improvements of density-based clustering, i.e., DScale [Zhu, Ting, and Angelova2018] and DP [Rodriguez and Laio2014].
The rest of the paper is organised as follows. We reiterate Isolation Kernel, identify the shortcomings of using the tree-based method to induce Isolation Similarity, provide the proposed alternative that employs a nearest neighbour method, the lemma and proof of the characteristic of Isolation Similarity, and the investigation in using Isolation Similarity in density-based clustering.
The descriptions of existing works are framed in order to clearly differentiate from the contributions we made here.
Isolation Kernel/Similarity is first proposed by [Ting, Zhu, and Zhou2018] as a new similarity which can adapt to density structure of the given dataset, as opposed to commonly used data independent kernels such as Gaussian and Laplacian kernels.
In the classification context, Isolation Kernel has been shown to be an effective means to improve the accuracy of SVM, especially in datasets which have varied densities in the class overlap regions [Ting, Zhu, and Zhou2018]. This is achieved by simply replacing the commonly used data independent kernel such as Gaussian and Laplacian kernels with the Isolation Kernel.
In the context of SVM classifiers, Isolation Kernel[Ting, Zhu, and Zhou2018] has been shown to be more effective than existing approaches such as distance metric learning [Zadeh, Hosseini, and Sra2016, Wang and Sun2015], multiple kernel learning [Rakotomamonjy et al.2008, Gönen and Alpaydin2011]
and Random Forest kernel[Breiman2000, Davies and Ghahramani2014].
The characteristic of Isolation Kernel is akin to one aspect of human-judged similarity as discovered by psychologists [Krumhansl1978, Tversky1977], i.e., human will judge the two same Caucasians as less similar when compared in Europe (which have many Caucasians) than in Asia.
We restate the definition and kernel characteristic [Ting, Zhu, and Zhou2018] below.
be a dataset sampled from an unknown probability density function. Let denote the set of all partitions that are admissible under where each isolating partition isolates one data point from the rest of the points in a random subset , and .
For any two points ,
Isolation Kernel of and wrt is defined to be
the expectation taken over the probability distribution on all partitioning
is defined to be the expectation taken over the probability distribution on all partitioningthat both and fall into the same isolating partition :
where is the indicator function which outputs 1 if is true; otherwise, .
is estimated from a finite number of partitioningsas follows:
The characteristic of Isolation Kernel is: two points in a sparse region are more similar than two points of equal inter-point distance in a dense region, i.e.,
Characteristic of : and such that , satisfies the following condition:
where and are two subsets of points in sparse and dense regions of , respectively; and is the distance between and .
To get the above characteristic, the required property of the space partitioning mechanism is to create large partitions in the sparse region and small partitions in the dense region such that two points are more likely to fall into a same partition in a sparse region than two points of equal inter-point distance in a dense region.
tree-based isolation partitioning
Isolation Kernel [Ting, Zhu, and Zhou2018] employs isolation trees or iForest [Liu, Ting, and Zhou2008] to measure the similarity of two points because its space partitioning mechanism produces the required partitions which have volumes that are monotonically decreasing wrt the density of the local region.
Here we identify two shortcomings in using isolation trees to measure Isolation Similarity, i.e., each isolation tree (i) employs axis-parallel splits; and (ii) is an imbalanced tree.
Figure 1(a) shows an example partitioning due to axis-parallel splits of an isolation tree. The tree-based isolating partitions generally satisfy the requirement of small partitions in dense region and large partitions in sparse region.
However, it produced some undesirable effect, i.e., some partitions are always overextended for the first few splits close to the root of an imbalanced tree111Imbalanced trees are a necessary characteristic of isolation trees [Liu, Ting, and Zhou2008] for their intended purpose of detecting anomalies, where anomalies are expected to be isolated with few splits; and normal points can only be isolated using a large number of splits.. These are manifested as elongated rectangles in Figure 1(a).
While using balanced trees can be expected to overcome this problem, the restriction to hyper-rectangles remains due to the use of axis-parallel splits.
To overcome these shortcomings of isolation trees, we propose to use a nearest neighbour partitioning mechanism which creates a Voronoi diagram [Aurenhammer1991] where each cell is an isolating partition (i.e., isolating one point from the rest of the points in the given sample.) An example is provided in Figure 1(b). Note that these partitions also satisfy the requirement of small partitions in the dense region and large partitions in the sparse region. But they do not have the undesirable effect of elongated rectangles. We provide our implementation in the next section.
Instead of using trees in its first implementation [Ting, Zhu, and Zhou2018], we propose to implement Isolation Similarity using nearest neighbours.
Like the tree method, the nearest neighbour method also produces each model which consists of isolating partitions , given a subsample of points. Rather than representing each isolating partition as a hyper-rectangle, it is represented as a cell in a Voronoi diagram [Aurenhammer1991], where the boundary between two points is the equal distance from these two points.
While the Voronoi diagram is nothing new, its use in measuring similarity is new.
Using the same notations as used earlier, is now a Voronoi diagram, built by employing points in , where each isolating partition or Voronoi cell isolates one data point from the rest of the points in . We call the point which determines a cell as the cell centre.
Given a Voronoi diagram constructed from a sample of points, the Voronoi cell centred at is:
where is a distance function and we use as Euclidean distance in this paper.
For any two points , the nearest neighbour-induced Isolation Similarity of and wrt is defined to be the expectation taken over the probability distribution on all Voronoi diagrams that both and fall into the same Voronoi cell :
where denotes the probability.
The Voronoi diagram has the required property of the space partitioning mechanism to produce large partitions in a sparse region and small partitions in a dense region. This yields the characteristic of Isolation Similarity : two points in a sparse region are more similar than two points of equal inter-point distance in a dense region.
The use of nearest neighbours facilitates a proof of the above characteristic that was previously hampered by the use of trees. We provide the proof in the next section.
Lemma and Proof of the characteristic of Isolation Similarity
Let denote the density at point , a lemma based on definition 4 is given below:
(sparse region) and (dense region) such that , the nearest neighbour-induced Isolation Similarity has the characteristic that for implies
Sketch of the proof: (i) If two points fall into the same Voronoi cell, then the distances of these individual points to this cell centre must be shorter than those to every other cell centre (or at most equal to those to one other cell centre) in a Voronoi diagram formed by all these cell centres. (ii) In a subset of points, sampled from , used to form a Voronoi diagram, the probability of two points falling into the same Voronoi cell can then be estimated based on the condition stated in (i). (iii) The probability of two points of equal inter-point distance falling into the same Voronoi cell is a monotonically decreasing function wrt the density of the cell.
Let a local region covering both and as a ball centred at the middle between and having as the diameter of the ball. Assume that the density in is uniform and denoted as .
Let be the -neighbourhood of , i.e., . The probability of both and are in the same Voronoi cell is equivalent to the probability of a point being the nearest neighbour of both and wrt all other points in , i.e., the probability of selecting points which are all located outside the region , where .
To simplify notation, is omitted. Then the probability of can be expressed as follows:
where denotes the cardinality of .
Assume that is also uniformly distributed, having the same density
is also uniformly distributed, having the same density, the expected value of can be estimated as:
where denotes the volume of .
Thus, we have
In other words, the higher the density in the area around and , the smaller is, as the volume of is constant given and .
Given two pairs of points from two different regions but of equal interpoint distance as follows: (sparse region) and (dense region) such that .
Assume that data are uniformly distributed in both regions, and we sample from such that and . We have because the volume of is equal to that of for , independent of the density of the region.
Supposing that we choose a sufficient large sample size of which contains points from both and . When the data are uniformly distributed in and , based on Equation 5, we have
This means that and (in a dense region) are more like to be in different cells than and in (in a sparse region), as shown in Figure 1.
A simulation validating the above analysis is given in Figure 2. It compares and when from a sparse region and from a dense region with equal inter-point distance. Given a fixed or a fixed inter-point distance, properties observed from Figure 2 are given as follows:
The rate of decrease of is faster than that of . Thus reaches 0 earlier.
Isolation Dissimilarity and contour maps
To be consistent with the concept of distance as a kind of dissimilarity, we use Isolation Dissimilarity hereafter:
Isolation dissimilarity: .
Like norm, and . However, depends on the data distribution and how is implemented, not the geometric positions only.
We denote the nearest-neighbour-induced Isolation Dissimilarity -aNNE; and the tree-induced version -iForest. An example comparison of the contour maps produced the two dissimilarities are given in Figure 3. Note that the contour maps of depend on the data distribution, whereas that of is not. Also, comparing to , the other dissimilarity change slower in area far from the centre point and faster in area close to the centre point.
We examine the impact of Isolation Dissimilarity on density-based clustering in the rest of the paper. We describe the neighbourhood density function commonly used in the density-based clustering and its counterpart called neighbourhood mass function in the next section.
Neighbourhood density and mass functions
Here we define mass-connected clusters defined in terms of neighbourhood mass function:
Using an -neighbourhood mass estimator , mass-connectivity with threshold between and via a sequence of unique points from , i.e., is denoted as , and it is defined as:
The second line denotes direct connectivity between two neighbouring points when . The last two lines denote transitive connectivity when .
A mass-connected cluster , which has a mode , is a maximal set of points that are mass-connected with its mode, i.e., .
Note that density-connectivity and density-connected clusters are similarly defined in DBSCAN [Ester et al.1996] when is replaced with in the above two definitions. In other words, DBSCAN [Ester et al.1996] which uses detects density-connected clusters; whereas DBSCAN which uses detects mass-connected clusters.
The only difference between a density-connected cluster and a mass-connected cluster is the dissimilarity measure used in Equation 6. We called the DBSCAN procedure which employs : MBSCAN, since it detects mass-connected clusters rather than density-connected clusters.
Condition under which MBSCAN detects all mass-connected clusters
Let a valley between two cluster modes be the points having the minimum estimated , i.e., , along any path linking cluster modes and . A path between two points and is non-cyclic linking a sequence of unique points starting with and ending with where adjacent points lie in each other’s -neighbourhood: .
Because (unlike used in ) is adaptive to the density of local data distribution, it is possible to adjust and to yield an distribution such that all valley-points have close enough small values, if there exist such and .
In other words, for some data distributions , there exist some and such that the distribution of satisfies the following condition:
where is the largest of the minimum estimated along any path linking cluster modes and .
In data distributions , MBSCAN is able to detect all mass-connected clusters because a threshold can be used to breaks all paths between the modes by assigning regions with estimated less than to noise, i.e.,
An example that subsumes , derived from the same dataset, is shown in Figure 4, where a hard distribution in which DBSCAN fails to detect all clusters is shown in Figure 4(a); but MBSCAN succeeds 222Note that the above condition was first described in the context of using mass-based dissimilarity [Ting et al.2018]; but not in relation to mass-connected clusters. We have made the relation to mass-connected clusters more explicitly here..
In other words, the mass distribution afforded by is more flexible than the density distribution generated by which leads directly to MBSCAN’s enhanced cluster detection capability in comparison with DBSCAN, though both are using exactly the same algorithm, except the dissimilarity.
Figure 5 shows the change of neighbourhood function values wrt the change in their parameter for using and using -aNNE. This example shows that no exists which enables DBSCAN to detect all three clusters. This is because the line for Peak#3 (which is the mode of the sparse cluster) has values in-between those of the two valleys. In contrast, many settings of of can be used to detect all three clusters because the lines of the two valleys are lower than those of the three peaks.
|Number of datasets with the Best score||5||0||1||4||2||14|
|#wins/#draws/#loses wrt MBSCAN--aNNE||5/2/13||0/0/20||1/2/18||4/2/14||1/1/18||-|
The aim of the experiments is to compare the clustering performance of DBSCAN using different dissimilarities relative to that of the state-of-the-art density-based clustering algorithm DP [Rodriguez and Laio2014]. In addition to the three dissimilarity measures, i.e., , -iForest and -aNNE, two recent distance transformation method called ReScale [Zhu, Ting, and Carman2016] and DScale [Zhu, Ting, and Angelova2018] are also included. Note that DBSCAN using are denoted as MBSCAN, as they are mass-based clustering methods.
All algorithms used in our experiments are implemented in Matlab (the source code with demo can be obtained from https://github.com/cswords/anne-dbscan-demo). We produced the GPU accelerated versions of all implementations. The experiments ran on a machine having CPU: i5-8600k 4.30GHz processor, 8GB RAM; and GPU: GTX Titan X with 3072 1075MHz CUDA [Owens et al.2008] cores & 12GB graphic memory.
A total of 20 datasets333The artificial datasets are from http://cs.uef.fi/sipu/datasets/ [Gionis, Mannila, and Tsaparas2007, Zahn1971, Chang and Yeung2008, Jain and Law2005] except that the hard distribution dataset is from https://sourceforge.net/p/density-ratio/ [Zhu, Ting, and Carman2016], 5 high-dimensional data are from http://featureselection.asu.edu/datasets.php [Li et al.2016], and the rest of the datasets are from http://archive.ics.uci.edu/ml [Dheeru and Karra Taniskidou2017]. are used in the experiments. They are from three categories: 5 artificial datasets, 6 high-dimensional datasets, and 9 general datasets. They are selected because they represent diverse datasets in terms of data size, number of dimensions and number of clusters. The data characteristics of these datasets are shown in the first four columns of Table 1. All datasets are normalised using the - normalisation so that each attribute is in [0,1] before the experiments begin.
We compared all clustering results in terms of the best score [Rijsbergen1979] 444, where and are the precision and the recall for cluster , respectively. is preferred over other evaluation measures such as Purity [Manning, Raghavan, and Schütze2008] and Normalized Mutual Information (NMI) [Strehl and Ghosh2002] because these measures do not take into account noise points which are identified by a clustering algorithm. Based on these measures, algorithms can obtain a high clustering performance by assigning many points to noise, which can be misleading in a comparison. that is obtained from a search of the algorithm’s parameter. We search each parameter within a reasonable range. The ranges used for all algorithms/dissimilarities are provided in Table 2. Because used in MBSCAN is based on randomised methods, we report the mean score over 10 trials for each dataset.
|DP||Target cluster number|
|neighbourhood size in|
|MBSCAN||neighbourhood size in|
|DScale||neighbourhood size in|
|* parameter is required for ReScale only.|
|A search of 10 values with equal interval in the range.|
Table 1 shows that MBSCAN using -aNNE has the best performance overall. Its scores are the best on 14 out of 20 datasets. The closest contender DP has the best scores on 5 datasets only. In two other performance measures, MBSCAN using -aNNE has 13 wins, 2 draws and 5 losses against DP; and has higher average score too (0.867 versus 0.823).
One notable standout is on the high-dimensional datasets: MBSCAN using -aNNE has the largest gap in average score in comparison with other contenders among the three categories of datasets. With reference to DBSCAN, the gap is close to 0.3 score; even compare with the closest contender DP, the gap is 0.1 score. The superiority of -aNNE over -iForest is also highlighted on these high-dimensional datasets.
MBSCAN using -iForest wins over the original DBSCAN on all datasets except one (aggregation). This version of MBSCAN uplifted the clustering performance of DBSCAN significantly to almost the same level of DP.
A significance test is conducted over MBSCAN with -aNNE, MBSCAN with -iForest and DP. Figure 6 shows the result of the test—MBSCAN using -aNNE performs the best and is significantly better than DP and MBSCAN using -iForest; and there is no significant difference between DP and MBSCAN using -iForest.
DBSCAN is known to be sensitive to its parameter settings. As MBSCAN is using exactly the same algorithm, it has same sensitivity.
A caveat is in order. Although Isolation Similarity consistently outperforms distance measure in DBSCAN, our preliminary experiment using DP shows that the result is mixed. An analysis of DP, similar to that provided in this paper, is required in order to ascertain the condition(s) under which DP performs well and Isolation Similarity can help.
Complexity and runtime
MBSCAN with -aNNE is much faster than MBSCAN with -iForest because the time complexity to build a maximum-size isolation tree and testing one point is ; and aNNE takes . The space complexity to store trained aNNE model is ; and that of iForest is .
Table 3 shows the GPU runtime results on the four largest datasets. In contrast to DBSCAN and DP, MBSCAN needs to pre-compute the dissimilarity matrix in the pre-processing step, and this takes time. This pre-processing constitutes the most of the time of MBSCAN reported in Table 3. -aNNE is still faster than ReScale in high-dimensional datasets, though it is one order of magnitude slower than DBSCAN and DP.
Summary: aNNE versus iForest implementations of Isolation Similarity
We find that the aNNE implementation of Isolation Similarity is better than the iForest implementation because the aNNE implementation is more:
Effective on datasets with varied densities and high-dimensional datasets.
Amenable to GPU acceleration because aNNE can be implemented in almost pure matrix manipulations. Thus, aNNE runs many orders of magnitude faster than iForest if a large is required because aNNE in the GPU implementation has almost constant runtime wrt .
Stable because aNNE’s randomisation is a result of sampling data subsets only.
We make four contributions in this paper:
Identifying shortcomings of tree-induced Isolation Similarity; proposing a nearest neighbour-induced Isolation Similarity to overcome these shortcomings; and establishing three advantages of the nearest neighbour-induced Isolation Similarity over the tree-induced one.
Formally proving the characteristic of the nearest neighbour-induced Isolation Similarity. This is the first proof since the introduction of Isolation Kernel [Ting, Zhu, and Zhou2018].
Providing a formal definition of mass-connected clusters and an explanation why detecting mass-connected clusters is a better approach in overcoming the shortcoming of DBSCAN (which detects density-connected clusters) in datasets with varied densities. This differs fundamentally from the existing density-based approaches of the original DBSCAN, DP and ReScale which all employ a distance measure to compute density.
Conducting an empirical evaluation to validate the advantages of (i) nearest-neighbour-induced Isolation Similarity over tree-induced Isolation Similarity; and (ii) mass-based clustering using Isolation Similarity over four density-based clustering algorithms, i.e., DBSCAN, DP, ReScale and DScale.
In addition, we show for the first time that it is possible to uplift the clustering performance of the classic DBSCAN, through the use of nearest-neighbour-induced Isolation Similarity, to surpass that of DP—the state-of-the-art density-based clustering algorithm.
This material is based upon work supported by eSolutions of Monash University (Xiaoyu Qin); and partially supported by the Air Force Office of Scientific Research, Asian Office of Aerospace Research and Development (AOARD) under award number: FA2386-17-1-4034 (Kai Ming Ting).
- [Aurenhammer1991] Aurenhammer, F. 1991. Voronoi diagrams—A survey of a fundamental geometric data structure. ACM Computing Surveys 23(3):345–405.
- [Borg, Groenen, and Mair2012] Borg, I.; Groenen, P. J.; and Mair, P. 2012. Applied multidimensional scaling. Springer Science & Business Media.
- [Breiman2000] Breiman, L. 2000. Some infinity theory for predictor ensembles. Technical Report 577. Statistics Dept. UCB.
[Chang and Yeung2008]
Chang, H., and Yeung, D.-Y.
Robust path-based spectral clustering.Pattern Recognition 41(1):191–203.
- [Davies and Ghahramani2014] Davies, A., and Ghahramani, Z. 2014. The random forest kernel and creating other kernels for big data from random partitions. arXiv:1402.4293.
- [Dheeru and Karra Taniskidou2017] Dheeru, D., and Karra Taniskidou, E. 2017. UCI machine learning repository.
- [Ester et al.1996] Ester, M.; Kriegel, H.-P.; Sander, J.; and Xu, X. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, volume 96, 226–231.
- [Gionis, Mannila, and Tsaparas2007] Gionis, A.; Mannila, H.; and Tsaparas, P. 2007. Clustering aggregation. ACM Transactions on Knowledge Discovery from Data (TKDD) 1(1):4.
- [Gönen and Alpaydin2011] Gönen, M., and Alpaydin, E. 2011. Multiple kernel learning algorithms. Journal Machine Learning Research 12:2211–2268.
- [Jain and Law2005] Jain, A. K., and Law, M. H. 2005. Data clustering: A user’s dilemma. In International conference on pattern recognition and machine intelligence, 1–10. Springer.
- [Krumhansl1978] Krumhansl, C. L. 1978. Concerning the applicability of geometric models to similarity data: The interrelationship between similarity and spatial density. Psychological Review 85(5):445–463.
- [Li et al.2016] Li, J.; Cheng, K.; Wang, S.; Morstatter, F.; Robert, T.; Tang, J.; and Liu, H. 2016. Feature selection: A data perspective. arXiv:1601.07996.
- [Liu, Ting, and Zhou2008] Liu, F. T.; Ting, K. M.; and Zhou, Z.-H. 2008. Isolation forest. In The Eighth IEEE International Conference on Data Mining, 413–422. IEEE.
- [Manning, Raghavan, and Schütze2008] Manning, C. D.; Raghavan, P.; and Schütze, H. 2008. Introduction to Information Retrieval. New York, NY, USA: Cambridge University Press.
- [Owens et al.2008] Owens, J. D.; Houston, M.; Luebke, D.; Green, S.; Stone, J. E.; and Phillips, J. C. 2008. Gpu computing. Proceedings of the IEEE 96(5):879–899.
- [Rakotomamonjy et al.2008] Rakotomamonjy, A.; Bach, F. R.; Canu, S.; and Grandvalet, Y. 2008. SimpleMKL. Journal of Machine Learning Research 9(Nov):2491–2521.
- [Rijsbergen1979] Rijsbergen, C. J. V. 1979. Information Retrieval. Newton, MA, USA: Butterworth-Heinemann, 2nd edition.
- [Rodriguez and Laio2014] Rodriguez, A., and Laio, A. 2014. Clustering by fast search and find of density peaks. Science 344(6191):1492–1496.
- [Steinbach, Ertöz, and Kumar2004] Steinbach, M.; Ertöz, L.; and Kumar, V. 2004. The challenges of clustering high dimensional data. In New directions in statistical physics. Springer. 273–309.
- [Strehl and Ghosh2002] Strehl, A., and Ghosh, J. 2002. Cluster ensembles—a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research 3(Dec):583–617.
- [Ting et al.2016] Ting, K. M.; Zhu, Y.; Carman, M.; Zhu, Y.; and Zhou, Z.-H. 2016. Overcoming key weaknesses of distance-based neighbourhood methods using a data dependent dissimilarity measure. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1205–1214. ACM.
- [Ting et al.2018] Ting, K. M.; Zhu, Y.; Carman, M.; Zhu, Y.; Washio, T.; and Zhou, Z.-H. 2018. Lowest probability mass neighbour algorithms: relaxing the metric constraint in distance-based neighbourhood algorithms. Machine Learning, doi.org/10.1007/s10994–018–5737–x.
- [Ting, Zhu, and Zhou2018] Ting, K. M.; Zhu, Y.; and Zhou, Z.-H. 2018. Isolation kernel and its effect on SVM. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2329–2337. ACM.
- [Tversky1977] Tversky, A. 1977. Features of similarity. Psychological Review 84(4):327–352.
- [Wang and Sun2015] Wang, F., and Sun, J. 2015. Survey on distance metric learning and dimensionality reduction in data mining. Data Mining and Knowledge Discovery 29(2).
- [Zadeh, Hosseini, and Sra2016] Zadeh, P.; Hosseini, R.; and Sra, S. 2016. Geometric mean metric learning. In International Conference on Machine Learning, 2464–2471.
- [Zahn1971] Zahn, C. T. 1971. Graph-theoretical methods for detecting and describing gestalt clusters. IEEE Transactions on computers 100(1):68–86.
- [Zhu, Ting, and Angelova2018] Zhu, Y.; Ting, K. M.; and Angelova, M. 2018. A distance scaling method to improve density-based clustering. In Phung, D.; Tseng, V. S.; Webb, G. I.; Ho, B.; Ganji, M.; and Rashidi, L., eds., Advances in Knowledge Discovery and Data Mining, 389–400. Cham: Springer International Publishing.
- [Zhu, Ting, and Carman2016] Zhu, Y.; Ting, K. M.; and Carman, M. J. 2016. Density-ratio based clustering for discovering clusters with varying densities. Pattern Recognition 60:983–997.