Incremental Method for Spectral Clustering of Increasing Orders

12/23/2015 ∙ by Pin-Yu Chen, et al. ∙ University of Michigan Indiana University IUPUI 0

The smallest eigenvalues and the associated eigenvectors (i.e., eigenpairs) of a graph Laplacian matrix have been widely used for spectral clustering and community detection. However, in real-life applications the number of clusters or communities (say, K) is generally unknown a-priori. Consequently, the majority of the existing methods either choose K heuristically or they repeat the clustering method with different choices of K and accept the best clustering result. The first option, more often, yields suboptimal result, while the second option is computationally expensive. In this work, we propose an incremental method for constructing the eigenspectrum of the graph Laplacian matrix. This method leverages the eigenstructure of graph Laplacian matrix to obtain the K-th eigenpairs of the Laplacian matrix given a collection of all the K-1 smallest eigenpairs. Our proposed method adapts the Laplacian matrix such that the batch eigenvalue decomposition problem transforms into an efficient sequential leading eigenpair computation problem. As a practical application, we consider user-guided spectral clustering. Specifically, we demonstrate that users can utilize the proposed incremental method for effective eigenpair computation and determining the desired number of clusters based on multiple clustering metrics.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 14

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Over the past two decades, the graph Laplacian matrix and its variants have been widely adopted for solving various research tasks, including graph partitioning [23], data clustering [14], community detection [28, 5], consensus in networks [19], dimensionality reduction [2], entity disambiguation [33], link prediction [32], graph signal processing [27], centrality measures for graph connectivity [4], interconnected physical systems [24], network vulnerability assessment [7], image segmentation [26], among others. The fundamental task is to represent the data of interest as a graph for analysis, where a node represents an entity (e.g., a pixel in an image or a user in an online social network) and an edge represents similarity between two multivariate data samples or actual relation (e.g., friendship) between nodes [14]. More often the eigenvectors associated with the smallest eigenvalues of the graph Laplacian matrix are used to cluster the entities into clusters of high similarity. For brevity, throughout this paper we will call these eigenvectors as the smallest eigenvectors.

The success of graph Laplacian matrix based methods for graph partitioning and spectral clustering can be explained by the fact that acquiring smallest eigenvectors is equivalent to solving a relaxed graph cut minimization problem, which partitions a graph into clusters by minimizing various objective functions including min cut, ratio cut or normalized cut [14]. Generally, in clustering is selected to be much smaller than

(the number of data points), making full eigen decomposition (such as QR decomposition) unnecessary. An efficient alternative is to use methods that are based on power iteration, such as Arnoldi method or Lanczos method, which computes the leading eigenpairs through repeated matrix vector multiplication. ARPACK

[13] library is a popular parallel implementation of different variants of Arnoldi and Lanczos method, which is used by many commercial software including Matlab.

However, in most situations the best value of is unknown and a heuristic is used by clustering algorithms to determine the number of clusters, e.g., fixing a maximum number of clusters and detecting a large gap in the values of the largest sorted eigenvalues or normalized cut score [21, 16]. Alternatively, this value of can be determined based on domain knowledge [1]. For example, a user may require that the largest cluster size be no more than 10% of the total number of nodes or that the total inter-cluster edge weight be no greater than a certain amount. In these cases, the desired choice of cannot be determined a priori

. Over-estimation of the upper bound

on the number of clusters is expensive as the cost of finding eigenpairs using the power iteration method grows rapidly with . On the other hand, choosing an insufficiently large value for runs the risk of severe bias. Setting to the number of data points is generally computationally infeasible, even for a moderate-sized graph. Therefore, an incremental eigenpair computation method that effectively computes the -th smallest eigenpair of graph Laplacian matrix by utilizing the previously computed smallest eigenpairs is needed. Such an iterative method obviates the need to set an upper bound on , and its efficiency can be explained by the adaptivity to increments in .

By exploiting the special matrix properties and graph characteristics of a graph Laplacian matrix, we propose an efficient method for computing the -th eigenpair given all of the smallest eigenpairs, which we call the Incremental method of Increasing Orders (Incremental-IO). For each increment, given the previously computed smallest eigenpairs, we show that computing the next smallest eigenpair is equivalent to computing a leading eigenpair of a particular matrix, which transforms potentially tedious numerical computation (such as the iterative tridiagonalization and eigen-decomposition steps in the Lanczos algorithm [11]) to simple matrix power iterations of known computational efficiency [10]. We then compare the performance of Incremental-IO with a batch computation method which computes all of the smallest eigenpairs in a single batch, and an incremental method adapted from the Lanczos algorithm, which we call the Lanczos method of Increasing Orders (Lanczos-IO). For a given number of eigenpairs iterative matrix-vector multiplication of Lanczos procedure yields a set of Lanczos vectors (), and a tridiagonal matrix (), followed by a full eigen-decomposition of ( is a value much smaller than the matrix size). Lanczos-IO saves the Lanczos vectors that were obtained while eigenpairs were computed and use those to generate new Lanczos vectors for computing -th eigenpair.

Comparing to the batch method, our experimental results show that for a given order , Incremental-IO provides a significant reduction in computation time. Also, as increases, the gap between Incremental-IO and the batch approach widens, providing an order of magnitude speed-up. Experiments on real-life datasets show that the performance of Lanczos-IO is overly sensitive to the selection of augmented Lanczos vectors, a parameter that cannot be optimized a priori—for some of our experimental datasets, Lanczos-IO performs even worse than the batch method (see Sec. 6). Moreover, Lanczos-IO consumes significant amount of memory as it has to save the Lanczos vectors () for making the incremental approach realizable. In summary, Lanczos-IO, although an incremental eigenpair computation algorithm, falls short in terms of robustness.

To illustrate the real-life utility of incremental eigenpair computation methods, we design a user-guided spectral clustering algorithm which uses Incremental-IO. The algorithm provides clustering solution for a sequence of values efficiently, and thus enable a user to compare these clustering solutions for facilitating the selection of the most appropriate clustering.

The contributions of this paper are summarized as follows.

  1. We propose an incremental eigenpair computation method (Incremental-IO) for both unnormalized and normalized graph Laplacian matrices, by transforming the original eigenvalue decomposition problem into an efficient sequential leading eigenpair computation problem. Simulation results show that Incremental-IO generates the desired eigenpair accurately and has superior performance over the batch computation method in terms of computation time.

  2. We show that Incremental-IO is robust in comparison to Lanczos-IO, which is an incremental eigenpair method that we design by adapting the Lanczos method.

  3. We use several real-life datasets to demonstrate the utility of Incremental-IO. Specifically, we show that Incremental-IO is suitable for user-guided spectral clustering which provides a sequence of clustering results for unit increment of the number

    of clusters, and updates the associated cluster evaluation metrics for helping a user in decision making.

2 Related Works

2.1 Incremental eigenvalue decomposition

The proposed method (Incremental-IO) aims to incrementally compute the smallest eigenpair of a given graph Laplacian matrix. There are several works that are named as incremental eigenvalue decomposition methods [17, 9, 18, 8, 25]. However, these works focus on updating the eigenstructure of graph Laplacian matrix of dynamic graphs when nodes (data samples) or edges are inserted or deleted from the graph, which are different from incremental computation of eigenpairs of increasing orders.

2.2 Cluster Count Selection for Spectral Clustering

Many spectral clustering algorithms utilize the eigenstructure of graph Laplacian matrix for selecting number of clusters. In [21], a value that maximizes the gap between the -th and the -th smallest eigenvalue is selected as the number of clusters. In [16], a value

that minimizes the sum of cluster-wise Euclidean distance between each data point and the centroid obtained from K-means clustering on

smallest eigenvectors is selected as the number of clusters. In [31], the smallest eigenvectors of normalized graph Laplacian matrix are rotated to find the best alignment that reflects the true clusters. A model based method for determining the number of clusters is proposed in [22]. Note that aforementioned methods use only one single clustering metric to determine the number of clusters and often implicitly assume an upper bound on (namely ).

3 Incremental Eigenpair Computation for Graph Laplacian Matrices

3.1 Background

Throughout this paper bold uppercase letters (e.g., ) denote matrices and (or ) denotes the entry in -th row and -th column of , bold lowercase letters (e.g., or ) denote column vectors, denotes matrix or vector transpose, italic letters (e.g., or ) denote scalars, and calligraphic uppercase letters (e.g., or ) denote sets. The vector of ones (zeros) is denoted by (). The matrix

denotes an identity matrix and the matrix

denotes the matrix of zeros.

We use two symmetric matrices, and , to denote the adjacency and weight matrices of an undirected weighted simple graph with nodes and edges. if there is an edge between nodes and , and otherwise. is a nonnegative symmetric matrix such that if and if . Let denote the strength of node . Note that when , the strength of a node is equivalent to its degree. is a diagonal matrix with nodal strength on its main diagonal and the off-diagonal entries being zero.

The (unnormalized) graph Laplacian matrix is defined as

(1)

One popular variant of the graph Laplacian matrix is the normalized graph Laplacian matrix defined as

(2)

where . The -th smallest eigenvalue and its associated unit-norm eigenvector of are denoted by and , respectively. That is, the eigenpair of has the relation , and . The eigenvectors have unit Euclidean norm and they are orthogonal to each other such that if and if . The eigenvalues of are said to be distinct if . Similar notations are used for .

Graph Type / Laplacian Matrix Unnormalized Normalized
Connected Graphs
Lemma 1
Theorem 1
Corollary 1
Corollary 3
Disconnected Graphs
Lemma 2
Theorem 2
Corollary 2
Corollary 4
Table 1: Utility of the lemmas, corollaries, and theorems.

3.2 Theoretical foundations of the proposed method (Incremental-IO)

The following lemmas and corollaries provide the cornerstone for establishing the proposed incremental eigenpair computation method (Incremental-IO). The main idea is that we utilize the eigenspace structure of graph Laplacian matrix to inflate specific eigenpairs via a particular perturbation matrix, without affecting other eigenpairs. Incremental-IO can be viewed as a specialized Hotelling’s deflation method

[20] designed for graph Laplacian matrices by exploiting their spectral properties and associated graph characteristics. It works for both connected, and disconnected graphs using both normalized and unnormalized graph Laplacian matrices. For illustration purposes, in Table 1 we group the established lemmas, corollaries, and theorems under different graph types and different graph Laplacian matrices. Because of the page limit, the proofs of the established lemmas, theorems and corollaries are given in the extended version111http://arxiv.org/abs/1512.07349.

Lemma 1

Assume that is a connected graph and is the graph Laplacian with denoting the sum of the entries in the -th row of the weight matrix . Let and define Then the eigenpairs of satisfy for and .

Corollary 1

For a normalized graph Laplacian matrix , assume is a connected graph and let . Then for and .

Lemma 2

Assume that is a disconnected graph with connected components, and let , let , and let . Then for , for , and
.

Corollary 2

For a normalized graph Laplacian matrix , assume is a disconnected graph with connected components. Let , and let Then
for , for , and .

Remark 1

note that the columns of any matrix with an orthonormal transformation matrix (i.e., ) are also the largest eigenvectors of and in Lemma 2 and Corollary 2. Without loss of generality we consider the case .

3.3 Incremental method of increasing orders

Given the smallest eigenpairs of a graph Laplacian matrix, we prove that computing the -th smallest eigenpair is equivalent to computing the leading eigenpair (the eigenpair with the largest eigenvalue in absolute value) of a certain perturbed matrix. The advantage of this transformation is that the leading eigenpair can be efficiently computed via matrix power iteration methods [11, 13].

Let be the matrix with columns being the smallest eigenvectors of and let be the diagonal matrix with being its main diagonal. The following theorems show that given the smallest eigenpairs of , the -th smallest eigenpair of is the leading eigenvector of the original graph Laplacian matrix perturbed by a certain matrix.

Theorem 1

(connected graphs) Given and , assume that is a connected graph. Then the eigenpair
is a leading eigenpair of the matrix . In particular, if has distinct eigenvalues, then .

The next theorem describes an incremental eigenpair computation method when the graph is a disconnected graph with connected components. The columns of the matrix are the smallest eigenvectors of . Note that has a canonical representation that the nonzero entries of each column are a constant and their indices indicate the nodes in each connected component [14, 6], and the columns of are the smallest eigenvectors of with eigenvalue [6]. Since the smallest eigenpairs with the canonical representation are trivial by identifying the connected components in the graph, we only focus on computing the -th smallest eigenpair given smallest eigenpairs, where . The columns of the matrix are the -th to the -th smallest eigenvectors of and the matrix is the diagonal matrix with being its main diagonal. If , and are defined as the matrix with all entries being zero, i.e., .

Theorem 2

(disconnected graphs) Assume that is a disconnected graph with connected components, given , and , the eigenpair is a leading eigenpair of the matrix . In particular, if has distinct nonzero eigenvalues, then .

Following the same methodology for proving Theorem 1 and using Corollary 1, for normalized graph Laplacian matrices, let be the matrix with columns being the smallest eigenvectors of and let . The following corollary provides the basis for incremental eigenpair computation for normalized graph Laplacian matrix of connected graphs.

Corollary 3

For the normalized graph Laplacian matrix of a connected graph , given and , the eigenpair is a leading eigenpair of the matrix . In particular, if has distinct eigenvalues, then
.

For disconnected graphs with connected components, let with columns being the -th to the -th smallest eigenvectors of and let . Based on Corollary 2, the following corollary provides an incremental eigenpair computation method for normalized graph Laplacian matrix of disconnected graphs.

Corollary 4

For the normalized graph Laplacian matrix of a disconnected graph with connected components, given , and , the eigenpair is a leading eigenpair of the matrix . In particular, if has distinct eigenvalues, then
.

3.4 Computational complexity analysis

Here we analyze the computational complexity of Incremental-IO and compare it with the batch computation method. Incremental-IO utilizes the existing smallest eigenpairs to compute the -th smallest eigenpair as described in Sec. 3.3, whereas the batch computation method recomputes all eigenpairs for each value of . Both methods can be easily implemented via well-developed numerical computation packages such as ARPACK [13].

Following the analysis in [10], the average relative error of the leading eigenvalue from the Lanczos algorithm [11] has an upper bound of the order , where is the number of nodes in the graph and is the number of iterations for Lanczos algorithm. Therefore, when one sequentially computes from to smallest eigenpairs, for Incremental-IO the upper bound on the average relative error of smallest eigenpairs is since in each increment computing the corresponding eigenpair can be transformed to a leading eigenpair computation process as described in Sec. 3.3. On the other hand, for the batch computation method, the upper bound on the average relative error of smallest eigenpairs is since for the -th increment () it needs to compute all smallest eigenpairs from scratch. These results also imply that to reach the same average relative error for sequential computation of smallest eigenpairs, Incremental-IO requires iterations, whereas the batch method requires iterations. It is difficult to analyze the computational complexity of Lanczos-IO, as its convergence results heavily depend on the quality of previously generated Lanczos vectors.

4 Application: User-Guided Spectral Clustering with Incremental-IO

Based on the developed incremental eigenpair computation method (Incremental-IO) in Sec. 3, we propose an incremental algorithm for user-guided spectral clustering as summarized in Algorithm 1. This algorithm sequentially computes the smallest eigenpairs via Incremental-IO (steps 1-3) for spectral clustering and provides a sequence of clusters with the values of user-specified clustering metrics.

The input graph is a connected undirected weighted graph and we convert it to the reduced weighted graph to alleviate the effect of imbalanced edge weights. The entries of are properly normalized by the nodal strength such that . We then obtain the graph Laplacian matrix for and incrementally compute the eigenpairs of via Incremental-IO (steps 1-3) until the user decides to stop further computation.

Starting from clusters, the algorithm incrementally computes the -th smallest eigenpair of with the knowledge of the previous smallest eigenpairs via Theorem 1 and obtains matrix containing smallest eigenvectors. By viewing each row in as a -dimensional vector, K-means clustering is implemented to separate the rows in into clusters. For each increment, the identified clusters are denoted by , where is a graph partition with nodes and edges.

Input: connected undirected weighted graph
Output: clusters
Initialization: . . Flag .
               . .
               . .
while Flag do
     1. .
     2. Compute the leading eigenpair
         and set .
     3. Update smallest eigenpairs of by
          and .
     4. Perform K-means clustering on the rows of
         to obtain clusters .
     5. Compute user-specified clustering metrics.
     if user decides to stop then Flag
         Output clusters
     else
         Go back to step 1 with .
     end if
end while
Algorithm 1 Incremental algorithm for user-guided spectral clustering using Incremental-IO (steps 1-3)

In addition to incremental computation of smallest eigenpairs, for each increment the algorithm can also be used to update clustering metrics such as normalized cut, modularity, and cluster size distribution, in order to provide users with clustering information to stop the incremental computation process. The incremental computation algorithm allows users to efficiently track the changes in clusters as the number of hypothesized clusters increases.

Note that Algorithm 1 is proposed for connected graphs and their corresponding unnormalized graph Laplacian matrices. The algorithm can be easily adapted to disconnected graphs or normalized graph Laplacian matrices by modifying steps 1-3 based on the developed results in Theorem 2, Corollary 3 and Corollary 4.

5 Implementation

We implement the proposed incremental eigenpair computation method (Incremental-IO) using Matlab R2015a’s “eigs” function, which is based on ARPACK package [13]. Note that this function takes a parameter and returns leading eigenpairs of the given matrix. The function is implemented in Matlab with a Lanczos algorithm that computes the leading eigenpairs (the implicitly-restarted Lanczos method [3]). This Matlab function iteratively generates Lanczos vectors starting from an initial vector (the default setting is a random vector) with restart. Following Theorem 1, Incremental-IO works by sequentially perturbing the graph Laplacian matrix with a particular matrix and computing the leading eigenpair of the perturbed matrix (see Algorithm 1) by calling . For the batch computation method, we use to compute the desired eigenpairs from scratch as increases.

Input: real symmetric matrix , of initial Lanczos vectors , of augmented Lanczos vectors
Output: leading eigenpairs of
Initialization: Compute Lanczos vectors as columns of and the corresponding tridiagonal matrix of . Flag . . .
while Flag do
     1. Obtain the leading eigenpairs of .
         .
     2. Residual error
     while Residual error > Tolerence do
         2-1.
         2-2. Based on and , compute the next
               Lanczos vectors as columns of and
               the augmented tridiagonal matrix
         2-3. and
         2-4. Go back to step 1
     end while
     3. . .
     if user decides to stop then Flag
         Output leading eigenpairs
     else
         Go back to step 1 with .
     end if
end while
Algorithm 2 Lanczos method of Increasing Orders (Lanczos-IO)

For implementing Lanczos-IO, we extend the Lanczos algorithm of fixed order ( is fixed) using the PROPACK package [12]. As we have stated earlier, Lanczos-IO works by storing all previously generated Lanczos vectors and using them to compute new Lanczos vectors for each increment in . The general procedure of computing leading eigenpairs of a real symmetric matrix using Lanczos-IO is described in Algorithm 2. The operation of Lanczos-IO is similar to the explicitly-restarted Lanczos algorithm [29], which restarts the computation of Lanczos vectors with a subset of previously computed Lanczos vectors. Note that the Lanczos-IO consumes additional memory for storing all previously computed Lanczos vectors when compared with the proposed incremental method in Algorithm 1, since the function uses the implicitly-restarted Lanczos method that spares the need of storing Lanczos vectors for restart.

To apply Lanczos-IO to spectral clustering of increasing orders, we can set to obtain the smallest eigenvectors of . Throughout the experiments the parameters in Algorithm 2 are set to be and , where is the machine precision, is the operator norm of , and these settings are consistent with the settings used in function [13]. The number of augmented Lanczos vectors is set to be , and the effect of on the computation time is discussed in Sec. 6.2. The Matlab implementation of the aforementioned batch method, Lanczos-IO, and Incremental-IO are available at https://sites.google.com/site/pinyuchenpage/codes.

6 Experimental Results

We perform several experiments: first, compare the computation time between Incremental-IO, Lanczos-IO, and the batch method; second, numerically verify the accuracy of Incremental-IO; third, demonstrate the usages of Incremental-IO for user-guided spectral clustering. For the first experiment, we generate synthetic Erdos-Renyi random graphs of various sizes. For the second experiment, we compare the consistency of eigenpairs obtained from Incremental-IO and the batch method. For the third experiment, we use six popular graph datasets as summarized in Table 2.

6.1 Comparison of computation time on simulated graphs

To illustrate the advantage of Incremental-IO, we compare its computation time with the other two methods, the batch method and Lanczos-IO, for varying order and varying graph size . The Erdos-Renyi random graphs that we build are used for this comparison. Fig. 1 (a) shows the computation time of Incremental-IO, Lanczos-IO, and the batch computation method for sequentially computing from to smallest eigenpairs. It is observed that the computation time of Incremental-IO and Lanczos-IO grows linearly as increases, whereas the computation time of the batch method grows superlinearly with .

Fig. 1 (b) shows the computation time of all three methods with respect to different graph size . It is observed that the difference in computation time between the batch method and the two incremental methods grow exponentially as increases, which suggests that in this experiment Incremental-IO and Lanczos-IO are more efficient than the batch computation method, especially for large graphs. It is worth noting that although Lanczos-IO has similar performance in computation time as Incremental-IO, it requires additional memory allocation for storing all previously computed Lanczos vectors.

Dataset Nodes Edges Density
Minnesota
Road222http://www.cs.purdue.edu/homes/dgleich/nmcomp/matlab/minnesota
2640
intersections
3302
roads
0.095%
Power
Grid333http://www-personal.umich.edu/ mejn/netdata
4941
power stations
6594
power lines
0.054%
CLUTO444http://glaros.dtc.umn.edu/gkhome/views/cluto
7674
data points
748718
kNN edges
2.54%
Swiss
Roll555http://isomap.stanford.edu/datasets.html
20000
data points
81668
kNN edges
0.041%
Youtube666http://socialcomputing.asu.edu/datasets/YouTube
13679
users
76741
interactions
0.082%
BlogCatalog777http://socialcomputing.asu.edu/datasets/BlogCatalog
10312
bloggers
333983
interactions
0.63%
Table 2: Statistics of datasets
(a)  
(b)  
Figure 1:

Sequential eigenpair computation time on Erdos-Renyi random graphs with edge connection probability

. The marker represents averaged computation time of 50 trials and the error bar represents standard deviation. (a) Computation time with

and different number of eigenpairs . (b) Computation time with and different number of nodes .
Figure 2: Computation time improvement of Incremental-IO relative to the batch method. Incremental-IO outperforms the batch method for all cases, and has improvement w.r.t. .
Figure 3: Computation time improvement of Lanczos-IO relative to the batch method. Negative values mean that Lanczos-IO requires more computation time than the batch method.

6.2 Comparison of computation time on real-life datasets

Fig. 3 shows the time improvement of Incremental-IO relative to the batch method for the real-life datasets listed in Table 2

, where the difference in computation time is displayed in log scale to accommodate large variance of time improvement across datasets that are of widely varying size. It is observed that the gain in computational time via Incremental-IO is more pronounced as cluster count

increases, which demonstrates the merit of the proposed incremental method.

On the other hand, although Lanczos-IO is also an incremental method, in addition to the well-known issue of requiring memory allocation for storing all Lanczos vectors, the experimental results show that it does not provide performance robustness as Incremental-IO does, as it can perform even worse than the batch method for some cases. Fig. 3 shows that Lanczos-IO actually results in excessive computation time compared with the batch method for four out of the six datasets, whereas in Fig. 3 Incremental-IO is superior than the batch method for all these datasets, which demonstrates the robustness of Incremental-IO over Lanczos-IO. The reason of lacking robustness for Lanczos-IO can be explained by the fact the previously computed Lanczos vectors may not be effective in minimizing the Ritz approximation error of the desired eigenpairs. In contrast, Incremental-IO and the batch method adopt the implicitly-restarted Lanczos method, which restarts the Lanczos algorithm when the generated Lanczos vectors fail to meet the Ritz approximation criterion, and may eventually lead to faster convergence. Furthermore, Fig. 4 shows that Lanczos-IO is overly sensitive to the number of augmented Lanczos vectors , which is a parameter that cannot be optimized a priori.

(a) Minnesota Road
(b) Power Grid
(c) CLUTO
(d) Swiss Roll
(e) Youtube
(f) BlogCatalog
Figure 4: The effect of number of augmented Lanczos vectors of Lanczos-IO in Algorithm 2 on computation time improvement relative to the batch method. Negative values mean that the computation time of Lanczos-IO is larger than that of the batch method. The results show that Lanczos-IO is not a robust incremental eigenpair computation method. Intuitively, small may incur many iterations in the second step of Algorithm 2, whereas large may pose computation burden in the first step of Algorithm 2, and therefore both cases lead to the increase in computation time.

Theorem 1 establishes that the proposed incremental method exactly computes the -th eigenpair using 1 to -th eigenpairs, yet for the sake of experiments with real datasets, we have computed the normed eigenvalue difference and eigenvector correlation of the smallest eigenpairs using the batch method and Incremental-IO as displayed in Fig. 6. The smallest eigenpairs are identical as expected; to be specific, using Matlab library, on the Minnesota road dataset for , the normed eigenvalue difference is and the associated eigenvectors are identical up to differences in sign. Results for the other datasets are reported in the extended version11footnotemark: 1.

6.3 Clustering metrics for user-guided spectral clustering

In real-life, an analyst can use Incremental-IO for clustering along with a mechanism for selecting the best choice of starting from . To demonstrate this, in the experiment we use five clustering metrics that can be used for online decision making regarding the value of . These metrics are commonly used in clustering unweighted and weighted graphs and they are summarized as follows.

1. Modularity: modularity is defined as

(3)

where is the set of all nodes in the graph, is the -th cluster, () denotes the sum of weights of all internal (external) edges of the -th cluster, , and denotes the total nodal strength.

2. Scaled normalized cut (SNC): NC is defined as [30]

(4)

SNC is NC divided by the number of clusters, i.e., NC/.

3. Scaled median (or maximum) cluster size: Scaled medium (maximum) cluster size is the medium (maximum) cluster size of clusters divided by the total number of nodes of a graph.

4. Scaled spectrum energy: scaled spectrum energy is the sum of the smallest eigenvalues of the graph Laplacian matrix divided by the sum of all eigenvalues of , which can be computed by

(5)

where is the -th smallest eigenvalue of and
is the sum of diagonal elements of .

These metrics provide alternatives for gauging the quality of the clustering method. For example, Mod and NC reflect the trade-off between intracluster similarity and intercluster separation. Therefore, the larger the value of Mod, the better the clustering quality, and the smaller the value of NC, the better the clustering quality. Scaled spectrum energy is a typical measure of cluster quality for spectral clustering [21, 16, 31], and smaller spectrum energy means better separability of clusters. For Mod and scaled NC, a user might look for a cluster count such that the increment in the clustering metric is not significant, i.e., the clustering metric is saturated beyond such a . For scaled median and maximum cluster size, a user might require the cluster count to be such that the clustering metric is below a desired value. For scaled spectrum energy, a user might look for a noticeable increase in the clustering metric between consecutive values of .

Figure 5: Consistency of eigenpairs computed by the batch computation method and Incremental-IO for Minnesota Road dataset.
Figure 6: Five clustering metrics computed incrementally via Algorithm 1 for Minnesota Road.

6.4 Demonstration

Here we use Minnesota Road data to demonstrate how users can utilize the clustering metrics in Sec. 6.3 to determine the number of clusters. The five metrics evaluated for Minnesota Road clustering with respect to different cluster counts are displayed in Fig. 6. Starting from clusters, these metrics are updated by the incremental user-guided spectral clustering algorithm (Algorithm 1) as increases. If the user imposes that the maximum cluster size should be less than of the total number of nodes, then the algorithm returns clustering results with a number of clusters of or greater. Inspecting the modularity one sees it saturates at , and the user also observes a noticeable increase in scaled spectrum energy when . Therefore, the algorithm can be used to incrementally generate four clustering results for , and . The selected clustering results in Fig. 7 are shown to be consistent with geographic separations of different granularity.

(a)
(b)
(c)
(d)
Figure 7: Visualization of user-guided spectral clustering on Minnesota Road with respect to selected cluster count . Colors represent different clusters.

We also apply the proposed incremental user-guided spectral clustering algorithm (Algorithm 1) to Power Grid, CLUTO, Swiss Roll, Youtube, and BlogCatalog. In Fig.8, we show how the value of clustering metrics changes with , for each dataset. The incremental method enables us to efficiently generate all clustering results with and so on. Due to space limitation, for each dataset we only show the trend of the three clustering metrics that exhibit the highest variation for different ; thus, the chosen clustering metrics can be different for different datasets. This suggests that selecting the correct number of clusters is a difficult task and a user might need to use different clustering metrics for a range of values, and Incremental-IO is an effective tool to support such an endeavor.

(a) Power Grid
(b) CLUTO
(c) Swiss Roll
(d) Youtube
(e) BlogCatalog
Figure 8: Three selected clustering metrics of each dataset. The complete clustering metrics can be found in the extended version (available at http://arxiv.org/abs/1512.07349).

7 Conclusion

In this paper we present Incremental-IO, an efficient incremental eigenpair computation method for graph Laplacian matrices which works by transforming a batch eigenvalue decomposition problem into a sequential leading eigenpair computation problem. The method is elegant, robust and easy to implement using a scientific programming language, such as Matlab. We provide analytical proof of its correctness. We also demonstrate that it achieves significant reduction in computation time when compared with the batch computation method. Particularly, it is observed that the difference in computation time of these two methods grows exponentially as the graph size increases.

To demonstrate the effectiveness of Incremental-IO, we also show experimental evidences that obtaining such an incremental method by adapting the existing leading eigenpair solvers (such as, the Lanczos algorithm) is non-obvious and such efforts generally do not lead to a robust solution.

Finally, we demonstrate that the proposed incremental eigenpair computation method (Incremental-IO) is an effective tool for a user-guided spectral clustering task, which effectively updates clustering results and metrics for each increment of the cluster count.

Acknowledgments

This work is partially supported by the Consortium for Verification Technology under Department of Energy National Nuclear Security Administration award number DE-NA0002534, by ARO grant W911NF-15-1-0479, by ARO grant W911NF-15-1-0241, and by Mohammad Hasan’s NSF CAREER Award (IIS-1149851).

References

  • [1] S. Basu, A. Banerjee, and R. J. Mooney. Active semi-supervision for pairwise constrained clustering. In SDM, volume 4, pages 333–344, 2004.
  • [2] M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
  • [3] D. Calvetti, L. Reichel, and D. C. Sorensen. An implicitly restarted lanczos method for large symmetric eigenvalue problems. Electronic Transactions on Numerical Analysis, 2(1), 1994.
  • [4] P.-Y. Chen and A. Hero. Deep community detection. IEEE Trans. Signal Process., 63(21):5706–5719, Nov. 2015.
  • [5] P.-Y. Chen and A. Hero. Phase transitions in spectral community detection. IEEE Trans. Signal Process., 63(16):4339–4347, Aug 2015.
  • [6] P.-Y. Chen and A. O. Hero. Node removal vulnerability of the largest component of a network. In GlobalSIP, pages 587–590, 2013.
  • [7] P.-Y. Chen and A. O. Hero. Assessing and safeguarding network resilience to nodal attacks. IEEE Commun. Mag., 52(11):138–143, Nov. 2014.
  • [8] C. Dhanjal, R. Gaudel, and S. Clémençon. Efficient eigen-updating for spectral graph clustering. Neurocomputing, 131:440–452, 2014.
  • [9] P. Jia, J. Yin, X. Huang, and D. Hu. Incremental laplacian eigenmaps by preserving adjacent information between data points. Pattern Recognition Letters, 30(16):1457–1463, 2009.
  • [10] J. Kuczynski and H. Wozniakowski. Estimating the largest eigenvalue by the power and lanczos algorithms with a random start. SIAM journal on matrix analysis and applications, 13(4):1094–1122, 1992.
  • [11] C. Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards, 45(4), 1950.
  • [12] R. M. Larsen. Computing the svd for large and sparse matrices. SCCM, Stanford University, June, 16, 2000.
  • [13] R. B. Lehoucq, D. C. Sorensen, and C. Yang. ARPACK users’ guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods, volume 6. Siam, 1998.
  • [14] U. Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4):395–416, Dec. 2007.
  • [15] R. Merris. Laplacian matrices of graphs: a survey. Linear Algebra and its Applications, 197-198:143–176, 1994.
  • [16] A. Y. Ng, M. I. Jordan, and Y. Weiss.

    On spectral clustering: Analysis and an algorithm.

    In NIPS, pages 849–856, 2002.
  • [17] H. Ning, W. Xu, Y. Chi, Y. Gong, and T. S. Huang. Incremental spectral clustering with application to monitoring of evolving blog communities. In SDM, pages 261–272, 2007.
  • [18] H. Ning, W. Xu, Y. Chi, Y. Gong, and T. S. Huang. Incremental spectral clustering by efficiently updating the eigen-system. Pattern Recognition, 43(1):113–127, 2010.
  • [19] R. Olfati-Saber, J. Fax, and R. Murray. Consensus and cooperation in networked multi-agent systems. Proc. IEEE, 95(1):215–233, 2007.
  • [20] B. N. Parlett. The symmetric eigenvalue problem, volume 7. SIAM, 1980.
  • [21] M. Polito and P. Perona. Grouping and dimensionality reduction by locally linear embedding. In NIPS, 2001.
  • [22] L. K. M. Poon, A. H. Liu, T. Liu, and N. L. Zhang. A model-based approach to rounding in spectral clustering. In UAI, pages 68–694, 2012.
  • [23] A. Pothen, H. D. Simon, and K.-P. Liou. Partitioning sparse matrices with eigenvectors of graphs. SIAM journal on matrix analysis and applications, 11(3):430–452, 1990.
  • [24] F. Radicchi and A. Arenas. Abrupt transition in the structural formation of interconnected networks. Nature Physics, 9(11):717–720, Nov. 2013.
  • [25] G. Ranjan, Z.-L. Zhang, and D. Boley. Incremental computation of pseudo-inverse of laplacian. In Combinatorial Optimization and Applications, pages 729–749. Springer, 2014.
  • [26] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 22(8):888–905, 2000.
  • [27] D. Shuman, S. Narang, P. Frossard, A. Ortega, and P. Vandergheynst.

    The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains.

    IEEE Signal Process. Mag., 30(3):83–98, 2013.
  • [28] S. White and P. Smyth. A spectral clustering approach to finding communities in graph. In SDM, volume 5, pages 76–84, 2005.
  • [29] K. Wu and H. Simon. Thick-restart lanczos method for large symmetric eigenvalue problems. SIAM Journal on Matrix Analysis and Applications, 22(2):602–616, 2000.
  • [30] M. J. Zaki and W. M. Jr. Data Mining and Analysis: Fundamental Concepts and Algorithms. Cambridge University Press, 2014.
  • [31] L. Zelnik-Manor and P. Perona. Self-tuning spectral clustering. In NIPS, pages 1601–1608, 2004.
  • [32] B. Zhang, S. Choudhury, M. A. Hasan, X. Ning, K. Agarwal, and S. P. andy Paola Pesantez Cabrera.

    Trust from the past: Bayesian personalized ranking based link prediction in knowledge graphs.

    In SDM MNG Workshop, 2016.
  • [33] B. Zhang, T. K. Saha, and M. A. Hasan. Name disambiguation from link data in a collaboration graph. In ASONAM, pages 81–84, 2014.

8 Supplementary File

8.1 Proof of Lemma 1

Since is a positive semidefinite (PSD) matrix, for all . Since is a connected graph, by (1) . Therefore, by the PSD property we have . Moreover, since is a symmetric real-valued square matrix, from (1) we have

(6)

By the PSD property of , we have since for any connected graph. Therefore, by the orthogonality of eigenvectors of (i.e., for all ) the eigenvalue decomposition of can be represented as

(7)

where and
for .

8.2 Proof of Lemma 2

The graph Laplacian matrix of a disconnected graph consisting of connected components can be represented as a matrix with diagonal block structure, where each block in the diagonal corresponds to one connected component in [6], i.e.,

(8)

where is the graph Laplacian matrix of -th connected component in . From the proof of Lemma 1 each connected component contributes to exactly one zero eigenvalue for , and

(9)

Therefore, we have the results in Lemma 2.

8.3 Proof of Corollary 1

Recall from (2) that , and also we have . Moreover, it can be shown that [15], and if is connected. Following the same derivation for Lemma 1 we obtain the corollary. Note that and .

8.4 Proof of Corollary 2

The results can be obtained by following the same derivation procedure in Sec. 8.2 and the fact that [15].

8.5 Proof of Theorem 1

From Lemma 1,

(10)

which is a valid eigenpair decomposition that can be seen by inflating the smallest eigenvalues of to with the originally paired eigenvectors. Using (8.5) we obtain the eigenvalue decomposition of as

(11)

Since , we have . Therefore, the eigenpair can be obtained by computing the leading eigenpair of . In particular, if has distinct eigenvalues, then the leading eigenpair of is unique. Therefore, by (8.5) we have the relation

(12)

8.6 Proof of Theorem 2

First observe from (8) that has zero eigenvalues since each connected component contributes to exactly one zero eigenvalue for . Following the same derivation procedure in the proof of Theorem 1 and using Lemma 2, we have

(13)

Therefore, the eigenpair can be obtained by computing the leading eigenpair of . If has distinct nonzero eigenvalues (i.e, ), we obtain the relation .

8.7 Complete clustering metrics for user-guided spectral clustering

Fig. 9 displays the five clustering metrics introduced in Sec. 6.3 on Minnesota Road, Power Grid, CLUTO, Swiss Roll, Youtube, and BlogCatalog. These metrics are computed incrementally via Algorithm 1 and they provide multiple criterions for users to decide the final cluster count.

8.8 Demonstration of equivalence between the proposed incremental computation method and the batch computation method

Here we use the smallest eigenpairs of the datasets in Table 2 computed by the batch method and Incremental-IO to verify Theorem 1, which proves that Incremental-IO exactly computes the -th eigenpair using 1 to -th eigenpairs. Fig. 10 shows the normed eigenvalue differences (in terms of root mean squared error) and the correlations of the associated eigenvectors obtained from the two methods. It can be observed that the normed eigenvalue difference is negligible and the associated eigenvectors are identical up to the difference in sign, i.e., the eigenvector correlation in magnitude equals to 1 for every pair of corresponding eigenvectors of the two methods.

(a) Minnesota Road
(b) Power Grid
(c) Swiss Roll
(d) CLUTO
(e) Youtube
(f) BlogCatalog
Figure 9: Multiple clustering metrics of different datasets listed in Table 2. The metrics are modularity, scaled normalized cut (NC/), scaled median and maximum cluster size, and scaled spectrum energy.
(a) Minnesota Road
(b) Power Grid
(c) CLUTO
(d) Swiss Roll
(e) Youtube
(f) BlogCatalog
Figure 10: Comparisons of the smallest eigenpairs by the batch method and Incremental-IO for datasets listed in Table 2.