# Graph Approximation and Clustering on a Budget

We consider the problem of learning from a similarity matrix (such as spectral clustering and lowd imensional embedding), when computing pairwise similarities are costly, and only a limited number of entries can be observed. We provide a theoretical analysis using standard notions of graph approximation, significantly generalizing previous results (which focused on spectral clustering with two clusters). We also propose a new algorithmic approach based on adaptive sampling, which experimentally matches or improves on previous methods, while being considerably more general and computationally cheaper.

## Authors

• 20 publications
• 59 publications
• 23 publications
• ### Spectral Clustering: An empirical study of Approximation Algorithms and its Application to the Attrition Problem

Clustering is the problem of separating a set of objects into groups (ca...
11/14/2012 ∙ by B. Cung, et al. ∙ 0

• ### Scalable Spectral Clustering Using Random Binning Features

Spectral clustering is one of the most effective clustering approaches t...
05/25/2018 ∙ by Lingfei Wu, et al. ∙ 0

• ### Unsupervised spectral learning

In spectral clustering and spectral image segmentation, the data is part...
07/04/2012 ∙ by Susan Shortreed, et al. ∙ 0

• ### Approximating Spectral Clustering via Sampling: a Review

Spectral clustering refers to a family of unsupervised learning algorith...
01/29/2019 ∙ by Nicolas Tremblay, et al. ∙ 0

• ### CLEAR: A Consistent Lifting, Embedding, and Alignment Rectification Algorithm for Multi-Agent Data Association

A fundamental challenge in many robotics applications is to correctly sy...
02/06/2019 ∙ by Kaveh Fathian, et al. ∙ 4

• ### Impact of regularization on Spectral Clustering

The performance of spectral clustering can be considerably improved via ...
12/05/2013 ∙ by Antony Joseph, et al. ∙ 0

• ### Clustering with t-SNE, provably

t-distributed Stochastic Neighborhood Embedding (t-SNE), a clustering an...
06/08/2017 ∙ by George C. Linderman, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Many unsupervised learning algorithms, such as spectral clustering

[18], [2] and low-dimensional embedding via Laplacian eigenmaps and diffusion maps [3],[16], need as input a matrix of pairwise similarities between the different objects in our data. In some cases, obtaining the full matrix can be a costly matter. For example, may be based on some expensive-to-compute metric such as W2D [5] ; based on some physical measurement (such as in some computational biology applications); or is given by a human annotator. In such cases, we would like to have a good approximation of the (initially unknown) matrix, while querying only a limited number of entries. An alternative but equivalent viewpoint is the problem of approximating an unknown weighted undirected graph, by querying a limited number of edges.

This question has received previous attention in works such as [17] and [9]

, which focus on the task of spectral clustering into two clusters, and assuming two such distinct clusters indeed exist (i.e. that there is a big gap between the second and third eigenvalues of the Laplacian matrix). In this work we consider, both theoretically and algorithmically, the question of query-based graph approximation more generally, obtaining results relevant beyond two clusters and beyond spectral clustering.

When considering graph approximations, the first question is what notion of approximation to consider. One important notion is cut approximation [14]

where we wish for every cut in the approximated graph to have weight close to the weight of the cut in the original graph up to a multiplicative factor. Many machine learning algorithms (and many more general algorithms) such as cut based clustering

[11], energy minimization [22], etc. [1] are based on cuts, so this notion of approximation is natural for these uses. A stronger notion is spectral approximation [19], where we wish to uniformly approximate the quadratic form defined by the Laplacian up to a multiplicative factor. This approximation is important for algorithms such as spectral clustering [2], Laplacian eigenmaps [3], diffusion maps [16], etc. that use the connection between the spectral properties of the Laplacian matrix and the graph.

Our theoretical analysis focuses on the number of queries needed for such approximations. We first consider the simple and intuitive strategy of sampling edges uniformly at random, and obtain results for both cut and spectral approximations, under various assumptions. We note that these results are considerably more general than the theoretical analysis in [17]

, which focuses on the behavior of the 2nd eigenvector of the Laplacian matrix, and crucially rely on this large eigengap. We then consider how to extend these results to adaptive sampling strategies, and design a generic framework as well as a new adaptive sampling algorithm for clustering (

CLUS2K). Compared to previous approaches, the algorithm is much simpler and avoids doing a costly full eigen-decomposition at each iteration, yet experimentally appears to obtain equal or even better performance on a range of datasets.

Our theoretical results build on techniques for graph sparsification ([14], [19]), where the task is to find a sparse approximation to a given graph . This is somewhat similar to our task, but with two important differences: First and foremost, we do not have access to the full graph, whereas in graph sparsification the graph is given, and this full knowledge is used by algorithms for this task (e.g. using the sum of edge weights associated with each node). Second, our goal is to minimize the number of edge sampled, not the number of edges in the resulting graph (of course by sampling a smaller number of edges we will get a sparser graph). Notice that if we wish to end with a sparse graph, one can always use any graph sparsification technique on our resulting graph and get a graph with guarantees on sparsity.

## 2 A General Graph Approximation Guarantee

For simplicity we will consider all graphs as full weighted graphs (with zero weights at edges that do not exist) and so any graph will be defined by a set of vertices and a weight matrix . We will start with a few basic definitions.

###### Definition 2.1.

Let be a weighted graph and a subset of vertices, then the cut defined by , , is the sum of all the weights of edges that have exactly one endpoint in .

###### Definition 2.2.

Let and be two graphs on the same set of vertices. is an -cut approximation of if for any we have

###### Definition 2.3.

Let . The graph Laplacian is defined as where is a diagonal matrix with values . The normalized graph Laplacian is defined as .

The Laplacian holds much information about the graph [4]. One of the main connections of the Laplacian, and in particular the quadratic form it represents, to the graph is through the equation

 xTLGx=12n∑i,j=1Wij(xi−xj)2 (1)

When this is easily seen to be the value of the cut defined by . Many spectral graph techniques, such as spectral clustering, can be seen as a relaxation of such a discrete problem to .

###### Definition 2.4.

A graph is an spectral approximation of if

 ∀x∈Rn(1−ϵ)xTL~Gx≤xTLGx≤(1+ϵ)xTL~Gx (2)

We note that this is different than requiring (using the matrix 2-norm) as we can view it as a multiplicative error vs. an additive error term. In particular, it implies approximation of eigenvectors (using the min-max theorem [4]), which is relevant to many spectral algorithms, and includes the approximation of the 2nd eigenvector (the focus of the analysis in [17]) as a special case Moreover, it implies cut approximation (via equation 1), and is in fact strictly stronger (see [19] for a simple example of a cut approximation which is not a spectral approximation). Therefore, we will focus on spectral approximation in our theoretical results.

Our initial approximation strategy will be to uniformly at random sample a subset of edges, i.e. pick edges without replacement and construct a graph with weights for any and zero otherwise, where

is the probability any edge is sampled. It is easy to see that the

.

We begin by providing a bound on which ensures an -spectral approximation. It is based on an adaptation of the work in [19], in which the author considered picking each edge independently. This differs from our setting, where we are interested in picking edges without replacement, since in this case the probabilities of picking different edges are no longer independent. While this seems like a serious complication, it can be fixed using the notion of negative dependence:

###### Definition 2.5.

are said to be negatively dependent if for all disjoint subset and all nondecreasing functions and ,

 E[f(Xi,i∈I)g(Xj,j∈J)]≤E[f(Xi,i∈I)]E[g(Xj,j∈J)]

Intuitively, a group of random variables are negatively dependent if when some of them have a high value, the others are more probable to have lower values. If we pick edges uniformly, each edge that has been picked lowers the chances of the other edges to get picked, so intuitively the probabilities are negatively dependent. The edge picking probabilities have been shown to be indeed negatively dependent in [17].

An important application of negative dependence is that the Chernoff-Hoeffding bounds, which hold for sums of independent random variables, also hold for negatively dependent variables. See supplementary material for details.

We can now state the general spectral approximation theorem:

###### Theorem 2.1.

Let be a graph with weights and its approximation after sampling edges uniformly. Define as the second smallest eigenvalue of and . If then the probability that is not an -spectral approximation is smaller then .

###### Proof sketch.

The proof is based on an adaptation of part of theorem 6.1 from [19]. The two main differences are that we use negative dependence instead of independence, and a weighted graph instead of an unweighted graph. The proof uses the following lemma

###### lemma 2.1.

Let be the normalized Laplacian of with second eigenvalue . If then is an -spectral approximation for .

The next part is to bound using a modified version of the trace method [21] in order to bound this norm. See the supplementary material for more details. ∎

Stating the result in a simplified form, we have that if , then one gets an -approximation guarantee using sampled edges.

The main caveat of theorem 2.1 is that it only leads to a non-trivial guarantee when and is not too small. Most algorithms, such as spectral clustering, assume that the graph has relatively small eigenvalues, in the ideal case (more then one connected component) we even have . Unfortunately, we will now show that this is unavoidable, and that the bound above is essentially optimal (up to log factors) for graphs with bounded , i.e. expanders. In the next section, we will show how a few reasonable assumptions allows us to recover non-trivial guarantees even in the regime of small eigenvalues.

Since spectral approximation implies cut approximation, we will use this to find simple bounds on the number of edges needed for both approximations. We will show that a necessary condition for any approximation is that the minimal cut is not too small, the intuition being that that even finding a single edge (for connectedness) on that cut can be hard, and get a lower bound on the number of samples needed. For this we will need to following simple lemma (which follows directly from the linearity of expectation)

###### lemma 2.2.

Let be a finite set, and . If we pick a subset of size uniformly at random then

We will now use this to prove a lower bound on the number of edges sampled for binary weighted graphs (i.e. unweighted graph) .

###### Theorem 2.2.

Let G be an binary weighted graph with minimal cut weight . Assume was constructed by sampling edges, then for any , .

###### Proof.

Let be all the edges in a minimal cut and let be the edges sampled. Since the weights are binary the weight of this cut in is the number of edges in . From lemma 2.2 we know that . From the Markov’s inequality we get that . If then the intersection is empty and we do not have an -approximation for any proving . ∎

This theorem proves that in order to get any reasonable approximation with a small budget (at least with uniform sampling) the original graphs minimal cut cannot be too small and that samples are needed. Comparing this to theorem 2.1 (noticing ) we see that, for graphs with a lower bound on by sampling within a logarithmic factor of this lower bound is sufficient to ensure a good cut approximation.

## 3 Clusterable Graphs

Clustering algorithms assume a certain structure of the graph, generally they assume strongly connected components, i.e. the clusters, with weak connections between them (the precise assumptions vary from algorithm to algorithm). While this is a bad scenario for approximation, as this normally means a small minimal cut (and for spectral clustering a small ), we will show how approximation can be used (on the inner-cluster graphs) to obtain useful results. We give two results, one geared towards spectral approximation, and the other towards cut approximation.

### 3.1 Spectral Approximation

###### Definition 3.1.

Assume a graph consists of clusters, define as the block diagonal matrix consisting of the similarity scores between same-cluster elements.

 Win=⎡⎢ ⎢ ⎢ ⎢ ⎢⎣W10⋯00W2⋯0⋮⋮⋱⋮00⋯Wk⎤⎥ ⎥ ⎥ ⎥ ⎥⎦

and the off-diagonal elements.

###### Assumption 3.1.

Define , the smallest of all the second normalized eigenvalues over all . Assume for some constant .

where for some .

###### Assumption 3.3.

Assume that for some .

Assumption 3.1 implies well connected clusters, while assumption 3.2 excludes sparse, well-connected graphs, which we have already shown earlier to be hard to approximate. Assumption 3.3 essentially requires the between-cluster connections to be relatively weaker than the within-cluster connections.

Under these assumptions, the next theorem proves the clusters can be found.

###### Theorem 3.1.

Let P be the zero eigenspace of

corresponding to the clusters, the Laplacian of the graph we get by sampling edges for , and the space spanned by the first eigenvectors of . Under previous assumptions, .

We simplified the statement in order not to get overwhelmed by notation. is a diagonal matrix whose diagonal values correspond to the canonical angles between the subspaces and , and is a common way to measure distance between subspaces.

###### Proof sketch.

If , i.e. was spanned by eigenvectors of the full , then the theorem would be true by the sin-theta theorem [6] using our assumptions. We need to show that this theorem can be used with . The sin-theta theorem states that where the ”noise” factor, and the unnormalized second eigenvalue of the ”signal” factor. Using theorem 2.1 and our first two assumptions we can approximate each and use to show that . We now only need to show . This can be done using the matrix Chernoff inequality [20], by applying a result in [12] that shows how it can be adapted to sampling without replacements. We note that the result in [12] is limited to sampling without replacements as negative dependence has no obvious extension to random matrices. For further details see the supplementary material ∎

This gives us a tradeoff between the number of edges sampled and the error. The theoretical guarantee from the sin-theta theorem for the complete graph is so for we have the same guarantee as if we used the full graph. For large enough one can get as small as desired by using .

### 3.2 Cut Approximation

Cut based clustering, such as [11], have a different natural notion of ”clusterable”. We will assume nothing on eigenvalues, making this more general than the previous section.

###### Assumption 3.4.

Assume can be partitioned into clusters, within which the minimal cut is at least . Furthermore, assume that any cut separating between the clusters of (but does not split same cluster elements) is smaller then , and that .

These assumptions basically require the inner-cluster connections to be relatively stronger than between-cluster connections.

###### Theorem 3.2.

Let be a graph with weights and its approximation after observing edges. Under previous assumptions if then the cuts separating the clusters are smaller then any cut that cuts into one of the clusters.

###### Proof Sketch.

We can use cut approximation for the clusters themselves so . Using the Chernoff bound and union bound for the cuts between clusters, we get that none of them is greater then . See the supplementary material for full proof.∎

In the supplementary material, we provide a more in-depth analysis of cut approximation including an analog of theorem 2.1.

## 4 Adaptive Sampling and the Clus2K Algorithm

Theorem 2.2 states that, with uniform sampling and no prior assumptions of the graph structure, we need at least where is the weight of the smallest cut. What if we had an adaptive algorithm instead of just uniform sampling? It is easy to see that for some graphs the same (up to a constant) lower bound holds. Think of a graph with vertices, consisting of two cliques that have c randomly chosen edges connecting them. Let’s assume further that some oracle told us which vertex is in which clique, so any sensible algorithm would sample only edges connecting the cliques. As the edges are random, it would take tries just to hit one edge needed for any good approximation. However, in some cases an adaptive scheme can reduce the number of samples, as we now turn to discuss in the context of clustering.

Consider a similar toy problem - we have a graph which is known to consist of two connected components, each a clique of size and we wish to find these clusters. We can run the uniform sampling algorithm until we have only two connected components and return them. How many edges do we need to sample until we get only two connected components? If we look only at one clique, then basic results in random graph theory [8] show that with high probability, the number of edges added before we get a connected graph is which lower bounds the number of samples needed. To improve on this we can use an adaptive algorithm with the following scheme: at each iteration, pick an edge at random connecting the smallest connected component to some other connected component. At each step we have at least a probability of to connect two connected components. This is because there are nodes in the wrong cluster, and at least in the right cluster (since we pick the smallest connected component). Therefore with high probability the number of steps needed to decrease the number of connected components from to two is .

This argument leads us to consider adaptive sampling schemes, which iteratively sample edges according to a non-uniform distribution. Intuitively, such a distribution should place more weight on edges which may be more helpful in approximating the structure of the original graph. We first discuss how we can incorporate

arbitrary non-uniform distributions into our framework. We then propose a specific non-uniform distribution, motivated by the toy example above, leading to a new algorithm for our setting in the context of clustering.

One approach to incorporate non-uniform distributions is by unbiased sampling, where we re-scale the weights according to the sampling probability. This means that the weights are unbiased estimates of the actual weights. Unfortunately, this re-scaling is not easy to compute in general when sampling without replacement, as the probability of sampling an edge is a marginal distribution over all the algorithm’s possible trajectories. Sampling with replacement is much easier, since it only depends on the sampling probability in the current iteration. Moreover, as long as we sample only a small part of all edges, the risk of re-sampling an already-sampled edge is negligible. Finally, one can show that whatever the non-uniform distribution, a simple modification (adding with probability half a uniform sample) suffices for cut approximation to hold. Unfortunately, we found this approach to work poorly in practice, as it was unstable and oscillated between good and bad clustering long after a good clustering is initially found.

Due to these issues, we consider a biased sampling without replacement approach, where we mix the non-uniform distribution with a uniform distribution (as proposed earlier) on unseen edges, but do not attempt to re-scale of weights. More specifically, consider any adaptive sampling algorithm which picks an unseen edge at step with probability that depends on the graph seen so far. We will consider a modified distribution that with probability picks an unseen edge uniformly, and with probability picks it according to . While biased sampling can ruin approximation guarantees, in the clustering scenarios one can show similar results to theorem 3.2 (under stronger conditions) for any adaptive sampling scheme. The theoretical guarantees for adaptive sampling are in the supplementary material.

### 4.1 Clus2k Algorithm

We now turn to consider a specific algorithmic instantiation, in the context of clustering. Motivated by the toy example from earlier, we consider a non-uniform distribution which iteratively attempts to connect clusters in the currently-observed graph, by picking edges between them. These clusters are determined by the clustering algorithm we wish to use on the approximated graph, and are incrementally updated after each iteration. Inspired by the common practice (in computer vision) of over-segmentation, we use more clusters than the desired number of clusters

( in our case). Moreover, as discussed earlier, we mix this distribution with a uniform distribution. The resulting algorithm, which we denote as CLUS2K, appears as Algorithm 1 below.

For the setting of budget-constrained clustering, the two most relevant algorithms we are aware of is the algorithm of [17] (hereby denoted as S&T), and the IU_RED algorithm of [9]. These algorithms are somewhat similar to our approach, in that they interleave uniform sampling and a non-uniform sampling scheme. However, the sampling scheme is very different than ours, and focuses on finding the edge to which the derivative of the 2nd Laplacian eigenvector is most sensitive. This has two drawbacks. First, it is specifically designed for spectral clustering and the case of clusters, which is based on the 2nd Laplacian eigenvector. Extending this to more than clusters requires either recursive partitioning (which can be suboptimal), or considering sensitivity w.r.t. eigenvectors, and it is not clear what is the best way to do so. Second, computing eigenvector derivatives requires a full spectral decomposition at each iteration, which can be quite costly or impractical for large matrices. In contrast, our algorithm does not compute derivatives. Therefore, when used with spectral clustering methods, which require only the smallest eigenvectors, we have a significant gain.

It is possible to speed up implementation even further, in the context of spectral clustering. Since only a single edge is added per iteration, one can use the previously computed eigenvectors as an initial value for fast iterative eigenvector solvers (although restarting every couple of steps is advised). Another possible option is to pick several edges from this distribution at each step, which makes this process parallelizable.

## 5 Experiments

We tested our CLUS2K algorithm on several datasets, and compared it to the S&T and IU_RED discussed earlier (other alternatives were tested in [17] and shown to be inferior). It is important to note that S&T and IU_RED were designed specifically for and spectral clustering using the unnormalized Laplacian , while we also tested for various values of , and using the normalized Laplacian as well [18]. The IU_RED performed badly (perhaps because it relies substantially on the assumption) in these cases while S&T performed surprisingly well (yet still inferior to CLUS2K ). Clustering was measured by cluster purity. The purity of a single cluster is the percent of the most frequent class in the cluster. The purity of a clustering is a weighted average of its single cluster purity, weighted by the number of elements in each cluster. The purity shown is averaged over 5 runs.

### 5.1 Synthetic Data

The synthetic experiments were performed on two datasets - The two half circles dataset, and a dataset comprising of four well separated Gaussians, both experiments used unnormalized spectral clustering (see figure 5.1) using a gaussian weight matrix. The two half circles is a classic clustering dataset with clusters. The Gaussian dataset shows how the various algorithms handle an easy dataset, and IU_RED performs worse than uniform sampling in this case.

### 5.2 Real Data

We tested on three further datasets - the iris and glass UCI datasets (both with k>2 clusters) using a gaussian weight matrix, and the Caltech-7 dataset, a subset of the Caltech-101 images datasets with 7 clusters gathered by [15], using the similarity matrix suggested by [10]. We tested each dataset using both the normalized and unnormalized Laplacian for clustering. The results are presented in figure 2

Overall, the experiments show that the CLUS2K algorithm performs as good or better than previous algorithms for budget-constrained clustering, while being significantly computationally cheaper as well as more general.

## References

• [1] Graph cuts in vision and graphics: Theories. Handbook of Mathematical Models of Computer Vision, 2006.
• [2] M. I. Jordan A. Y. Ng and Y. Weiss.

On spectral clustering: Analysis and an algorithm.

Advances in Neural Information Processing Systems, 2001.
• [3] M. Belkin and P. Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. 2001.
• [4] F. R. K Chung. Spectral Graph Theory. American Mathematical Society, 1997.
• [5] C. Gollan D. Keysers, T. Deselaers and H. Ney. Deformation models for image recognition. Transactions on Pattern Analysis and machine Intelligence, 2007.
• [6] C. Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. Journal on Numerical Analysis, 1970.
• [7] D. Dubhashi and A. Panconesi. Concentration of Measure for the Analysis of Randomized Algorithms. Cambridge University Press, 2012.
• [8] P. Erdős and A. Rényi. On random graphs. Publicationes Mathematicae, pages 290–297, 1959.
• [9] N. Jojic F. Wauthier and M. Jordan. Active spectral clustering via iterative uncertainty reduction. SIGKDD Conference on Knowledge Discovery and Data Mining, 2012.
• [10] A. Faktor and M. Irani. clustering by composition unsupervised discovery of image categories. European Conference on Computer Vision, 2013.
• [11] R. Tarjan G. Flake and K.Tsioutsiouliklis. Graph clustering and minimum cut trees. Internet Mathematics, 2003.
• [12] D. Gross and V. Nesme. Note on sampling without replacing from a finite collection of matrices. CoRR, abs/1001.2738, 2010.
• [13] D. Karger. Global min-cuts in and other ramifications of a simple mincut algorithm. In SODA, 1993.
• [14] D. R. Karger. using randomized sparsification to approximate minimum cuts. In Proceedings of the 5th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 424–432, 1994.
• [15] Y.J. Lee and K. Grauman. Foreground focus: Unsupervised learning from partially matching. International Journal of Computer Vision, 2009.
• [16] B. Nadler, S. Lafon, R. Coifman, and I. Kevrekidis.

Diffusion maps, spectral clustering and eigenfunctions of fokker-planck operators.

In NIPS, 2005.
• [17] O. Shamir and N. Tishby. Spectral clustering on a budget. Journal of Machine Learning Research, 2011.
• [18] J. Shi and J. Malik. Normalized cuts and image segmentation. Transactions on Pattern Analysis and Machine Intelligance, 2000.
• [19] D. Spielman and S.-H. Teng. Spectral sparsifi cation of graphs. SIAM Journal on Computing, 40, 2011.
• [20] J. A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 2012.
• [21] V. Vu. Spectral norm of random matrices. Combinatorica,, 2007.
• [22] O. Veksler Y. Boykov and R. Zabih. Fast approximate energy minimization via graph cuts. Transactions on Pattern Analysis and machine Intelligence, 2001.

## Appendix A Proof of Theorem 2.1

We will need to Hoeffding-Chernoff bound for negative dependence:

###### Theorem A.1.

Assume are negatively dependent variables and define then

 P(|X−E[X]|≥ϵE[X])≤2exp(−ϵ2E[X]3)

See [7] 1.6 and 3.1 for details.

First it is important to note a change in notation from [19] in order to be consistent with notation used in [14]. A -spectral approximation in our paper is weaker then a -spectral approximation in [19].

We will now go over the main changes needed to prove Theorem 6.1 in [19] (disregarding S.2) with negatively dependent sampling of edges and weights .

The proof of Claim 6.5 is quite straightforward. The claim of Lemma 6.6 needs to be changed to instead of . The changes to the proof are again straightforward (remembering ).

The main change is to Lemma 6.4. Using the modified Lemma 6.6 and substituting negative dependents for independence one can prove

 ∑σvalidforT,τ∏s∈T⎡⎣Δvs−1,vs∏i:τ(i)=sΔvi−1,vi⎤⎦≤1γk−|T|∑σvalidforT,τ∏s∈Twvs−1,vsdvs−1.

instead of equation 10 in the paper. The last change is to pick proportional to instead of uniformly to prove that

 ∑σvalidforT,τ∏s∈Twvs−1,vsdvs−1≤1

instead of equation 11. From there on all changes are straightforward.

## Appendix B Proof of Theorem 3.1

Let . Let be the zeros eigenspace of , which is the same as the zero eigenspace of , if all the are connected. Let be the space spanned by the first eigenvectors of . According to the Sin-Theta theorem [6], where is the spectral norm of and is the second smallest unnormalized eigenvalue of . To prove the theorem we will show that and that .

The first claim is through using the first two assumptions and the following lemma

###### lemma B.1.

Let and be the second smallest normalized and unnormalized eigenvalues of , and then .

###### Proof.

From the min-max theorem we have

 μ2=minU:dim(U)=2{maxx∈U∖{0}xTLx||x||2}
 λ2=minU:dim(U)=2{maxx∈U∖{0}xTLxxTDx}

and the lemma follows from the fact that .∎

Let be the number of edges needed in order to have an -spectral approximation of each inner-cluster matrix for with probability , then by theorem 2.1 we have that . Using this fact, the first two assumptions and lemma B.1, it is easy to see that .

We now need to show that . The main tool would be the matrix Chernoff inequlity for sampling matrixes without replacements.

###### Theorem B.1.

Consider a finite sequence of Hermitian matrices sampled uniformly without replacements from a finite set of matrices of dimension n. Assume that

 Xk⪰0||Xi||≤R.

Define then

 P(||Y||≥(1+ϵ)||E[Y]||)≤n⋅(eϵ(1+ϵ)1+ϵ)||E[Y]||/R
###### Proof.

This is an adaptation of theorem 5.1.1 from [20] replacing the independence requirement to sampling without replacements. In order to adapt the proof we notice that the only place where independence is used in in lemma 3.5.1 (subadditivity of the matrix cumulant generating functions) where we need to prove that

 ∀θ∈RE[Tr(exp(∑θXi))]≤Tr(exp(∑logEeθXi)) (3)

Using the result of [12], if are sampled uniformly at random without replacements for a finite set, and are sampled with the same probability with replacements then

 E[Tr(exp(∑θXi))]≤E[Tr(exp(∑θYi))] (4)

so we can conclude that

 E[Tr(exp(∑θXi))]≤E[Tr(exp(∑θYi))]≤
 ≤Tr(exp(∑logEeθYi))=Tr(exp(∑logEeθXi)).

where the second inequality is from 3 as are independent.

We define for each edge connecting nodes in different clusters the matrix that is equal to zero with probability and is equal to with probability , where is the Laplacian of a single edge graph with weight . Then , , and .

If we use the matrix Chernoff inequality with then

 P(||~Lout||≥2enγ)≤n(12nγ−β)2enγp

So if we get that for large enough .

## Appendix C Cut Approximation

We will start by proving an analog of theorem 2.1 in the paper. We will use the following lemma from [13]:

###### lemma C.1.

Let be an undirected graph with vertices and minimal cut . For all the number of cuts with weight smaller of equal to is less then .

The lemma is proven in [13] for graphs with integer weights, but the extension to any positive weights is trivial by scaling and rounding. We can now state and prove the theorem guaranteeing good cut approximations.

###### Theorem C.1.

Let be a graph with weights , with minimal cut , and its approximation after sampling edges uniformly. If where , then the probability that is not an -cut approximation is smaller then .

###### Proof.

This is an adaptation of the proof in [14] - consider a cut with weight . Let the probability to sample a single edge. Let where is an indicator whether edge on the cut was sampled and its weight. Define the sum of on all the edges along the cut, then by the fact that edges are negatively dependent and theorem A.1, the probability that the cut is not an approximation is smaller then

 2exp(−ϵ2E[Y]3)=2exp(−ϵ2αcp3)≤≤2exp(−(ln(1δ)+k)α)⋅n−2α

Define and let the number of cuts with value in the original graph. By the union bound the probability that some cut is not an approximation is less then (notice that this sum is well defined since is non zero only in a finite number of values). Defining then by the previous lemma . Let be any measure on such that , then the integral is maximized when . This is due to the fact that is a monotonically decreasing function, so if the inequality is not tight at some point we could increase the value by picking for some appropriate and (where is the Dirac delta function) . From this we can conclude that the probability of some cut not being an -approximation is bounded by

 n2P(1)+∞∫1P(α)dn2αdαdα=2δe−k+4δln(n)e−kln(1δ)+k≤δ(2+4ln(n))exp(−k)=δ

A drawback is that the theorem gives a bound that depends on the minimal cut, which we do not know, and unlike the situation in [14] we cannot approximate it using the full graph. We can prove a bound that uses only known data about the graph. The following theorem shows we can lower bound .

###### lemma C.2.

Let be a graph with weights , with minimal cut , and its approximation after sampling edges with minimal cut . Define the probability to sample a single edge. Also define and . With probability greater then the following inequality holds - .

###### Proof.

Let be a subset of vertices such that then from the Chernoff-Hoeffding inequality (the one-sided version)

 P(|∂~GS|≥(1+ϵ)|∂GS|)=P(p|∂~GS|≥(1+ϵ)p|∂GS|)≤exp(−ϵ2pc3)

Where we multiply by to have all the elements bounded by . Setting we get that with probability greater then that
. By completing the square we get that

 ⎛⎜⎝√cp+√3ln(1δ)4⎞⎟⎠2=(√cp+√l)2≥p~c+l

which means (after some simple algebraic manipulation) that

 c≥~cβ2

We can combine these to theorems and get

###### Theorem C.2.

Let be a graph with weights and its approximation after sampling edges with minimal cut . Define and as in previous theorems. If then the probability that is not an -cut approximation is smaller then .

###### Proof.

This is just using lemma C.2 with error probability and using that for theorem C.1 with the same error probability and the union bound. ∎

This theorem gives a high probability bound that depends only on observable quantities. While the notation is a bit cumbersome, it is easy to see that if , i.e. the unscaled weight of the smallest cut is not too small, then and we have a bound that is almost as good as if we knew the real .

We will now prove theorem 3.2 in the paper.

###### Theorem (3.2).

Let be a graph with weights and its approximation after observing edges. Assume is partitioned into clusters each has minimal cut greater or equal to , and the cuts separating clusters from the others is smaller then . Furthermore assume . If then the cuts separating the clusters are smaller then any cut that cuts into one of the clusters.

###### Proof.

After seeing edges, the probability for sampling any edge inside any cluster is . By theorem C.1 we have that if then the probability of any cut in a single cluster being smaller then is smaller then , with the union bound we have that with probability greater then all cuts in any cluster (and therefore any cut in that cuts some cluster) have weights greater or equal to .

We now need to show that the cuts separating the clusters are not too large. Consider a cut separating some clusters from the others. If the weight of this cut is we need to show that with probability greater then we have . This means that we want to show that , i.e. we can use the negatively dependent Chernoff-Hoeffding inequality (theorem A.1) with (using the fact that ) and get that the . As we can finish the proof. ∎

## Appendix D Adaptive Sampling with replacement

While we found that adaptive sampling with replacements did not perform as well as without replacements in practice, for completeness we will present her a proof that it has the same theoretical guarantees as uniform sampling for cut approximation.

Let be the graph build at step , an adaptive sampling algorithm is an algorithm who picks an edge at step with probability that depends on . In order to prove that with high probability is a -approximation of for we need that isn’t too small on any edge. This can be easily done by sampling according to a modified distribution - with probability pick an edge uniformly, and with probability pick it according to . The new distribution satisfies .

The graphs are by no means independent. Although one can view (after subtracting the mean) them as a martingale process, using the method of bounded differences [7] will not suffice, as it depends on the square of the bounding constant, so we will have a factor that only gives a trivial bound. We will next show that a high probability bound does exists.

Consider a cut with weight that contains the edges and consider any bounded adaptive sampling algorithm with replacements with steps. Define with and to be the random variable that has value if the edge was chosen at step and zero otherwise. Define , is the weight added to the cut at step and its expectation is .

###### lemma D.1.

If and then

 E[exp(tρYk)|~Gk−1]≤exp(cρ(et−1))
###### Proof.

Since at most one of the positive variables is nonzero for a constant then they are negatively dependent when conditioned by . This implies that . By definition of we get that

 E[exp(tρXik)|~Gk−1]=~p(ei)⋅exp(tρw(ei)~p(ei))+(1−~p(ei)) (5)

One can easily verify that the right hand side of equation 5 decreases monotonically with , so the fact that and implies that

 E[exp(tρXik)|~Gk−1]≤ρw(ei)et+(1−ρw(ei))=
 =ρw(ei)(et−1)+1≤exp(ρw(ei)(et−1))

Where the last inequality is due to the fact that for . We can finish the proof since

 E[exp(tρYk)|~Gk−1]≤l∏i=1E[exp(tρXik)|~Gk−1]≤exp(ρc(et−1)).

as .∎

We can now prove the concentration of measure bound for a single cut

###### Theorem D.1.

Let be a graph such that and the output of a bounded adaptive sampling algorithm with replacements such that then the probability that a cut with weight in is not a -approximation is bounded by .

###### Proof.

We need to show that

 P(∣∣ ∣∣m∑k=1Yk−mc∣∣ ∣∣>ϵmc)≤2exp(−ϵ2ρmc3)

The proof is similar to the proof of the Chernoff bound, replacing independence with lemma D.1. First look at . Using the standard trick for all

 P(m∑k=1Yk>(1+ϵ)mc)=P(exp(tρm∑k=1Yk)>exp(t(1+ϵ)ρmc))

By the Markov inequality this is bounded by . The law of total expectation states that . As is a deterministic function of this is equal to

using lemma D.1. By induction we can conclude that the expectation is smaller then . We have shown that

 P(m∑k=1Yk>(1+ϵ)mc)≤exp(ρmc(et−1))exp(t(1+ϵ)ρmc)

Following the steps as in the standard Chernoff bound proof one can show that this is smaller (for the right ) then . The proof for this bound on is done in a similar fashion, and using the union bound we finish our proof.

Using one can now show similar theorems to what we shown in the previous section with this theorem replacing the (negatively dependent) Chernoff bound.

## Appendix E Adaptive Sampling without Replacements

For a specific graph one can always design a bad biased sampling scheme. Consider an adversarial scheme that always samples the largest weight edge between two constant clusters, it is easy to see that this can lead to bad cut clustering. To circumvent this we will consider graphs where the edge weights between the clusters, which we regard as noise, are picked randomly.

###### Assumption E.1.

Assume can be partitioned into clusters of size , within which the minimal cut is at least .

###### Assumption E.2.

Assume that the weights of edges between the clusters are , besides edges chosen uniformly at randomly (without replacement) between any two clusters that have weight .

###### Theorem E.1.

Let be the graph after sampling edges without replacements (with probability of sampling uniformly) with . Let and be the minimal cut weight inside any cluster and the maximal cut weight between clusters, under previous assumptions the probability that is smaller then .

###### Proof.

Using theorem C.1 on the edges sampled uniformly (remembering that the biased sampling can only increase the cut weight) we get that with probability greater then ,