Scalable Sparse Subspace Clustering via Ordered Weighted ℓ_1 Regression

07/10/2018 ∙ by Urvashi Oswal, et al. ∙ University of Wisconsin-Madison 0

The main contribution of the paper is a new approach to subspace clustering that is significantly more computationally efficient and scalable than existing state-of-the-art methods. The central idea is to modify the regression technique in sparse subspace clustering (SSC) by replacing the ℓ_1 minimization with a generalization called Ordered Weighted ℓ_1 (OWL) minimization which performs simultaneous regression and clustering of correlated variables. Using random geometric graph theory, we prove that OWL regression selects more points within each subspace, resulting in better clustering results. This allows for accurate subspace clustering based on regression solutions for only a small subset of the total dataset, significantly reducing the computational complexity compared to SSC. In experiments, we find that our OWL approach can achieve a speedup of 20× to 30× for synthetic problems and 4× to 8× on real data problems.



There are no comments yet.


page 15

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Subspace clustering refers to the task of grouping high-dimensional data points into distinct subspaces. This generalizes classical, single-subspace approaches to data modeling like principal components analysis (PCA). Effectively, subspace clustering aims to represent data in terms of a union of subspace (UoS). Many applications, ranging from computer vision (

e.g., image segmentation [21], motion segmentation [18] and face clustering [10]) to network analysis [6], have demonstrated the advantages of this generalization. Unlike classical PCA, the problem of fitting a UoS model to data is a computationally challenging task, and numerous approaches have been proposed. For a comprehensive review of these algorithms, we refer the reader to the tutorial [17]. The state-of-the-art is Sparse Subspace Clustering (SSC) [5] which provides both tractability and provable guarantees under mild conditions [14].

SSC is a computationally intensive method. It requires performing a sparse regression for each of the points in the dataset of interest. The main contribution of this paper is a new approach that has the potential to significantly reduce the computational complexity, making it more applicable to large-scale problems. The central idea is to modify the regression technique so that accurate clustering is possible using only the results of a regressions, instead of all . This reduces the complexity by a factor of . The modified regression is based on the Ordered Weighted (OWL) regularizer, which performs simultaneous regression and clustering of correlated variables. The clustering property of the OWL, combined with ideas from random geometric graph theory, allows us to prove that the new approach, called OWL Subspace Clustering (OSC), tends to select more points from the correct subspaces in each regression compared to SSC. In the ideal case, where subspaces are orthogonal and the number of points per subspace is sufficiently large, then OSC can succeed with just optimizations (gain factor of roughly ) as detailed later in Section 1.4. This key feature of OSC makes accurate clustering possible based on regression solutions for only a small subset of the total dataset, significantly reducing the computational complexity compared to SSC. In experiments, we find that OSC can achieve a speedup of 20 to 30 even for small scale synthetic problems.

1.1 Relation to prior work

When the data is high-dimensional or the number of data points is large, solving the Lasso problems (each in variables) in SSC can be computationally challenging. Greedy algorithms for computing sparse representations of the data points (in terms of all the other data points) have therefore been popular alternatives. Broadly speaking, three kinds of such algorithms have been proposed in the literature, namely Thresholded subspace clustering [8], subspace clustering using Orthogonal Matching Pursuit (OMP) [4, 22], and Dimensionality-reduced subspace clustering [19, 9]. TSC relies on the nearest neighbors–in spherical distance–of each data point to construct the adjacency matrix. SSC-OMP employs OMP instead of the Lasso to compute sparse representations of the data points. Similar in spirit to SSC-OMP, Nearest subspace neighbor (NSN) [12] greedily assigns to each data point a subset of the other data points by iteratively selecting the data point closest (in Euclidean distance) to the subspace spanned by the previously selected data points. In this paper, we propose OWL Subspace Clustering (OSC), which employs the Ordered Weighted (OWL) regularizer instead of the Lasso to compute sparse representations of the data points. The clustering property of OWL (discussed next) leads to a hybrid approach between SSC and spherical distance based methods like TSC (illustrated in Figure 1). The effect is that OSC tends to assign non-zero weights to the same points as SSC and additionally selects neighbors of these points in terms of Euclidean distance. All of the above methods, including OSC, can be carried out on dimensionality reduced data points, in the spirit of [19, 9]. We compare the performance of these methods in Section 5.

Figure 1: 2-dimensional subspaces in , and . The points with non-zero coefficients in solutions of and Ordered Weighted sparse regressions on (yellow point) depicted as black and green points (and edges) respectively. The automatic clustering property of OWL leads to a hybrid approach that tends to select same points as SSC and their Euclidean distance based neighbors.

1.2 Review of SSC

The key idea of SSC is representing each data point as a sparse linear combination of the remaining data points. The rationale is that points in the same subspace are likely to be selected to represent a given point, and thus the selected points provide an indication of the cluster. Sparse linear regression using

minimization is used to determine the representations for each point. SSC solves sparse linear regressions over data points to form an

adjacency matrix, which in turn defines a graph where the vertices are data points with edges indicated by the adjacencies. Spectral clustering is used to partition the graph, and hence the data, into clusters. Each sparse linear regression solves


where the columns of represent the -dimensional data points and

is a data point that is to be represented as a linear combination of the columns using the sparse coefficient vector

. Another sparse regression commonly used in practice is the Lasso given by


for . It is sometimes referred to as the Lagrangian or “noisy” version since it is used for subspace clustering with noise [20, 15].

1.3 OWL norm and its clustering property

A new family of regularizers, called Ordered weighted (OWL), for sparse linear regression was recently introduced and studied in [23, 1, 7, 2]. The authors show that OWL automatically clusters and averages regression coefficients associated with strongly correlated variables. This has a desirable effect of selecting more of the relevant variables than the penalty. The main idea in this paper is to leverage this feature of OWL for subspace clustering. OWL tends to select more points from the common subspace in each regression, compared to methods (see Figure 2), which consequently improves the performance of spectral clustering.

Figure 2: Solutions of the , , and OWL minimizations for lying in . 10 points selected from each of the three subspaces ordered such that the first and the last 10 points belong to and respectively. The solution corresponds to choosing two other points lying in whereas the solution selects points from all subspaces. OWL selects more points from than .

The OWL norm [23, 1] is defined as


where denotes the -th largest magnitude, is a vector of weights, such that and . The OWL norm reduces to the norm when all the weights are set to be equal and the norm when . It is also useful to look at the structure of the OWL norm balls. Figure 3 shows the OWL balls in .

Figure 3: OWL balls in with different choices of weights leading to the and as special cases. (Left) and . (Middle) and . (Right)

. Contours centered at Least squares estimate (

Low correlation (dotted), leads to solution at .
High correlation (solid), leads to clustered solution at .

The OWL-regularized regression used in this paper is


The following result provides a sufficient condition stating that when columns of are correlated enough, OWL automatically clusters them (i.e., correlated columns will have equal-valued coefficients in the solution). Let be the minimum gap between consecutive elements of .

Lemma 1 (Theorem 2.1 in [7])

Let be a solution of (4), and and be two columns of . If , then .

In this paper, we propose to replace the minimization in SSC with the OWL regularizer in order to discover many more points from the true subspace in each optimization. The key observation is that points in the same subspace will be more correlated (in the sense above), than points in different subspaces. We formalize this idea using tools from random geometric graph theory. Next, we demonstrate the benefit of the clustering property of OWL in a simple setting where the subspaces are orthogonal to each other.

1.4 Orthogonal subspaces example

To build some intuition, let be a matrix whose columns are drawn from a union of orthogonal linear subspaces, . Let be a new point from subspace . To keep the notation simple, let us assume that the dimension of each subspace is . Suppose contains the indices of columns belonging to subspace and .

It is easy to see that any solution of (4), , satisfies for all , since columns from orthogonal subspaces cannot reduce the residual error and including them will increase the OWL penalty. Additionally, assume that points in the subspace are uniformly drawn from the unit hypersphere in that subspace. Then consider the graph constructed by placing an edge between pairs of points (vertices) that are within Euclidean distance of each other (where is the minimum gap in the OWL weights). A simple argument (developed in Section G

) shows that the resulting random geometric graph is connected with probability at least

, if the number of points in the subspace is large enough, specifically . It then follows from Lemma 1 that for . In other words, all the columns within the subspace will be selected and have equal-valued coefficients.

Figure 4: Examples of coefficient matrices for exact minimizations (left) and OWL optimizations (right) with the contiguous columns lying in three orthogonal subspaces each of dimension in . The plots were generated using OWL-Ramp weights defined in Section 3.

Figure 4 shows an example of the coefficients generated by performing the optimizations in (1) and (4) on a collection of points from three orthogonal subspaces. The results obtained with orthogonal subspaces suggest that, in an ideal scenario, we could perform subspace clustering by running only one OWL optimization per subspace without the need for spectral clustering leading to a total of only optimizations where is the number of subspaces. This can easily be achieved by running one OWL optimization on a randomly selected point, removing the points chosen by OWL and repeating the process till no points remain. This works since OWL has the potential to find all the points in the same subspace in a single run. This has the potential of significantly reducing the computational complexity of SSC. However, OWL could lead to clustering of points from other subspaces when subspaces are not orthogonal. This tradeoff is studied in more detail in the rest of the paper by providing theoretical and empirical evidence suggesting that an OWL-based fast subspace clustering algorithm (stated in Section 2) can reduce the computational complexity of SSC by a factor depending on the nature of the subspaces.

2 Algorithm

The SSC procedure can easily be modified by replacing the sparse linear regression step with the OWL optimization in (4). As seen in the example with orthogonal subspaces, OWL solutions are denser due to its clustering property. This leads to an intuitive extension of the algorithm where we run only a subset of the total possible optimizations. By running a random subset of optimizations the computational complexity can be reduced. We formalize this version of OSC in Algorithm LABEL:owlalgo.


The intuition developed in the case of orthogonal subspaces suggests that taking , where is the number of subspaces may suffice for the OWL approach. Thus, the proposed algorithm requires solving sparse regressions, whereas traditional SSC involves solving , and typically . It is also worth noting that the computational complexity of lasso and OWL is essentially the same. Obviously, one could also consider reducing the computational complexity of traditional SSC by also using only regressions instead of all . However, as shown in our experiment section, SSC performance quickly degrades when the number of regressions is reduced, while the proposed OWL-based algorithm’s performance does not, leading to a speedup of up to 30 in some cases.

3 Theoretical results

In this section, we begin by formalizing the notations and model, followed by the main results analyzing the behavior of the solution of the optimization (4). We also provide examples and remarks to understand the theoretical results along with proof sketches. The proof sketches also provide the intermediate results ranging from sparse regression with correlated variables to random geometric graph theory used to prove the main results.

3.1 Notation and model

We are given data points lying in a union of unknown linear subspaces; there are subspaces of of dimensions . We are given a collection of data points as columns of , which may be partitioned as without loss of generality; for each is a collection of vectors that belong to subspace . The goal is to approximate the underlying subspaces using the points in . We also assume the columns are normalized to have unit norm, . The notation used is summarized in Table 1.

Number of subspaces
Dimension of each subspace for
Number of points sampled from each subspace
Sampling density
Total number of points,
Ambient dimension
component of OWL-Ramp,
Slope of OWL-Ramp,
Length of OWL-Ramp,
Normalized maximum affinity between subspaces,
Number of optimizations
Noise level
Table 1: Notation and parameters

We consider the intuitive semi-random model introduced in [14] where the subspaces are fixed, and points are distributed randomly on each of the subspaces. To measure the notion of closeness or correlation between two subspaces, the affinity between subspaces is used.

Definition 1

The principal angles between subspaces and of dimensions and , are defined by

with orthogonality constraints .

Definition 2

The normalized affinity between subspaces is

The affinity is low when the subspaces are nearly orthogonal and high when the subspaces overlap significantly (it is equal to one when one subspace is contained in the other). Hence, when the affinity is high, clustering is hard whereas it becomes easier as the affinity decreases.

3.2 Performance metrics

To quantify performance of the algorithm, we use the following metrics from [15]

False discovery: Fix and and let be the outcome of Step1 in Algorithm 1. Then we say that obeying is a false discovery if and do not belong to the same subspace.

True discovery: Similarly, we say that obeying is a true discovery if and belong to same subspace.

3.3 Main results

We state the first main result showing that the solution of the OWL optimization does not include false discoveries if the the affinity between the subspaces is small enough, i.e., the subspaces are not too close.

Let be average of the tail of the OWL weights used. If , the subspace to which the -th column belongs, obeys


where is a fixed numerical constant, then for all , i.e., there is no false discovery in the -th column of with probability at least . Roughly stated the result says that with high probability the OWL solution contains no false discoveries if the ratio of OWL parameters is big enough for fixed subspaces. The affinity is higher for overlapping subspaces, then we must reduce the gaps in the OWL weights ( in the introduction). This is roughly equivalent to making the ratio of weights nearly 1. As seen in the introduction, the clustering property of OWL depends on gaps in consecutive weights. OWL clusters columns that are at most the gap away from each other. Hence, making the weight ratio close to 1 results in reducing the weight gaps and in turn reduces the radius of OWL clustering. For smaller values of affinity, the OWL gap can be made bigger and OWL will then group (assign equal coefficients to) more points. Thus, there is a tension in the closeness of the subspaces and the radius or gap of OWL clusters. For higher values of affinity, OWL with bigger weight gaps does not provide any benefit over Lasso hence we must set the weight ratio to one. The weight ratio is exactly one for the penalty and our bound for the special case of Lasso matches [14] for SSC.

As foreshadowed in the introduction, we show that all the columns within the subspace of interest will have equal-valued coefficients if enough points are sampled, specifically . Recall that is the minimum gap in the OWL weights. Usually, this gap is very small, like O() or in some cases zero, which makes the requirement infeasible. Hence, as part of the second main result, we will prove new clustering bounds for a specific type of the OWL norm called OWL-Ramp with weights

where , and . Figure 5 depicts an example of the OWL-Ramp weights. The OSCAR regularizer [2] is a special case obtained by setting . Note that if , then and the affinity condition in Theorem 3.3 becomes

Figure 5: An example of OWL-Ramp weights.

Let be a solution of (4) with OWL-Ramp weights. If , the conditions in Theorem 3.3 are satisfied, and the


points within the subspace of interest are sampled uniformly at random from the unit hypersphere, then the set has cardinality greater than or equal to the ramp parameter, , with probability at least . ( is a fixed numeric constant.) The result roughly says that if enough points are sampled from the corresponding subspace then with high probability the top coefficients in the OWL solution have equal magnitude. Combining this with Theorem 3.3, if the solution to OWL optimization is non-trivial, this is equivalent to making true discoveries.

It is easy to see that , if where is the dual norm of OWL defined later in the section. In order to ensure that the solution is non-trivial we need at least for . The term scales at most like and for OWL-Ramp, . Intuitively, the component needs to be made small enough to achieve a non-trivial solution.

To make sense of the results, the condition in (5) can be rewritten as an upper bound on and the condition in (6) as a lower bound on . Ignoring constants and terms

Intuitively, we want to make small enough so that nothing from outside of the true subspace is selected, but at the same time we want to make as big as possible so many points are selected from the true subspace. This leads to a trade-off between the number of false discoveries and true discoveries. For orthogonal subspaces or , the upper bound is trivially satisfied and we can make big enough to group all the points in the same subspace as seen in the introduction. For or when the subspace is contained in another subspace, needs to be set to zero to ensure no false discoveries. Equivalently, the best we can do in this situation is the Lasso solution. Guided by the theory we state rules of thumb for selecting the parameters of OSC followed in the experiments of this paper.

3.4 Choice of hyper-parameters.

The and parameters are varied in Section 4 to demonstrate the trade off between the number of false discoveries and true discoveries. In some applications like motion segmentation, the dimensions of the subspaces are equal and known in advance or can be roughly estimated. In such cases we recommend setting the tuning parameters as follows.

  • Ramp length, : In general, we recommend setting proportional to an estimate of number points per cluster since this parameter is related to the number of points clustered by OWL.

  • component, : Informed by theory and experiments in this paper and from SSC literature[15], we set proportional to .

  • Ramp slope or gap, : Since the gap in the OWL weights control the clustering behavior the theory suggests setting for many true discoveries. As the affinity of the subspaces increases needs to be reduced in order to avoid false discoveries. In the experiments, we observe that even small values such as lead to enough clustering in OWL solutions given that there are enough points in each subspace.

In the rest of the section, we state the important intermediate results used to prove the main results and provide a sketch of the proofs. Proof details are provided in the Appendix. The intermediate results are stated in generality since they may be useful in other settings.

3.5 Proof of Theorem 3.3

We start by proving a deterministic lemma that introduces the OWL dual feasibility condition. First we define the dual norm of the OWL norm given by [23]

where is the sub-vector of , consisting of the largest magnitude elements of and .

Lemma 2

Fix and . Suppose is a solution to

obeying where . Then any optimal solution to

must also satisfy .

Proof details are provided in the Appendix. The lemma tells us that if the OWL dual feasibility condition is satisfied for , then we have no false discoveries. To prove Theorem 3.3, it suffices to show that the dual feasibility condition is satisfied. One sufficient condition to satisfy the OWL dual feasibility condition is since it can be shown easily that for , we have .

We want to show the OWL dual feasibility is satisfied. Using Theorem 7.5 in [14] stated in Appendix with details, we can show that,


We next prove a bound on the size of the residual where is defined in Lemma 2.

Lemma 3

If , then with probability at least , we have

Proof details are provided in the Appendix. Lemma 3 gives with high probability,

where , for . Finally using the assumption on the affinity of the subspaces along with union bound and Lemma 2 completes the proof.

3.6 Proof of Theorem 3.3

The points on the unit hypersphere can be viewed as forming a -dimensional Random Geometric Graph by placing an edge between points that are -close. To formalize this notion we define a -RGG on the unit hypersphere below (slightly different from the typical RGG defined in the literature [13, 3]).

Definition 3

-Random Geometric Graph (RGG): Place points uniformly at random on the surface of unit hypersphere . Place an edge between pairs of points (vertices) that are within Euclidean distance of each other.

It can be shown that if enough points are sampled uniformly at random from the unit sphere leads to a fully connected -RGG with high probability. The connectivity and percolation of a typical RGG has been studied ([13] and references therein) in the past but the proof of the main result hinges on a fully connected graph defined on the unit hypersphere. Hence, we derive a new bound next.

Figure 6: Random geometric graphs in showing effects of different values of and . A -hypersphere is drawn centered at every point. An edge is placed between pairs of points if the corresponding hyperspheres intersect. The graph connectivity grows as or increase.
Lemma 4

If points are sampled uniformly at random from the unit hypersphere in , then the -RGG formed by these points is fully connected with probability at least .

is a fixed numeric constant. To prove the result we use a covering argument to divide the surface of the unit hypersphere into equal area patches such that the distance between points in adjacent patches is at most , followed by a multiplicative form of Chernoff’s bound to show with high probability at least one point of the uniformly sampled points falls into each patch leading to a fully connected -RGG. Proof details are provided in the Appendix. Using the fact that the Euclidean norm is invariant under multiplication with orthonormal matrix, the distances between points within the subspace can be translated to the ambient subspace. Similarly, we can define a -RGG on the unit hypersphere in .

We prove a new clustering property for the OWL-Ramp regression next.

Lemma 5

Let be a solution of the optimization in (4) using OWL-Ramp weights (, and ) and . If elements of belong to a -connected component on the unit hypersphere in with cardinality at least , then .

Proof details are provided in the Appendix. This lemma provides a sufficient condition for the largest magnitude cluster in the OWL-Ramp solution to have critical mass. The Euclidean distance condition from Lemma 1 translates into the -connected component condition in this Lemma. Note that the minimum gap is always smaller than or equal to . It is strictly smaller in most cases. This provides more room for clustering in the OWL solution.

The proof of Theorem 3.3 follows by combining the results from Lemma 5 and Lemma 4 with Theorem 3.3 as explained in Appendix.

4 Numerical results

Figure 7: (a) Trade-off curves for FPR generated by sweeping through (, ) values. Empirical averages of (FPR,TPR) are shown for Exact , Lasso, and OWL, over random points. (b) Clustering error for varying in Algorithm LABEL:owlalgo

. The affinity matrix is generated for each method by running only a random subset

of the total optimizations.
Figure 8: Clustering error for varying in Algorithm LABEL:owlalgo with varying affinity (left) and number of points sampled (right) for OWL regularized subspace clustering.

In this section we present numerical results on synthetic data corroborating the theoretical guarantees and providing a better understanding of the behavior of OSC particularly we focus on the OWL-Ramp norm defined in Section 3.3. The subspaces and are generated with dimension in with ambient dimension .

Bases generation method, B1: The bases , and

are obtained by choosing, uniformly at random, from the set of all sets of orthonormal vectors in

. Since sum of the subspace dimensions exceeds ambient dimension, , this ensures that the subspaces overlap and leads to .

Bases generation method, B2: To generate subspaces of varying affinity, we follow the method described in [14]. The subspaces , and are generated using the bases

respectively. Where the principal angles are set in such a way that the normalized affinity decreases linearly from 1 to 0.75 and diag(cos ) corresponds to a diagonal matrix with diagonal entries equal to cos .

We select points per subspace dimension (unless stated otherwise), i.e, points uniformly at random from each subspace using the respective bases.

Choice of hyper-parameters. See Section 3.4 for discussion on the choice of tuning parameters , and guided by the theory in this paper and past literature. We fix the ramp parameter, , in the experiments.

The FPR-TPR trade off. In order to compare the performance of the optimizations in (1), (2) and (4), we generate columns from subspaces generated using B1 to run optimizations of each method. For each data point, we sweep through different values of the tuning parameters (). For , all points are selected and as is increased fewer points are given non-zero weight. Let denote the solution to one of the optimizations. We plot the empirical averages of False Positive Rate (FPR) and True Positive Rate (TPR) where is the part of supported on indices of the points from the same subspace and is supported on its complement. A non-zero entry in is a true discovery and likewise a non-zero entry in is a false discovery. By definition and .

Figure 7 (a) shows the Receiver Operating Characteristic (ROC) curve plotting TPR versus FPR. The solution of an exact minimization is shown as one point on the curve since the sparsity of the solution cannot be changed by varying tuning parameters. Note for any , the lasso solution will include at most non-zero entries, since the number of selected columns will not exceed the ambient dimension. If , then lasso will select all columns. Thus, lasso curve beyond selections is shown as a dashed line, which extends linearly to the point (FPR,TPR) = . As suggested by theory and demonstrated by the plots, for a fixed false positive rate, OWL can achieve a much higher () true positive rate than Lasso.

Effect of the size of the optimization subset . To observe the effect of the size of the optimization subset , we look at the clustering error produced by running a subset of the total optimizations for each method for different values of . The subspaces are generated using B1. We take the symmetrized affinity matrix, , generated by each method, assuming knowledge of number of subspaces, apply a spectral clustering method to obtain the clusters. The clustering error is measured as the fraction of misclassified points from the total number of points. Figure 7 (b) shows how the clustering error varies with for SSC with Exact , Lasso, and OWL regularized sparse linear regression steps. We report the empirical average of the clustering error over 100 random choices of subsets. By virtue of the clustering property of OWL, we see that the clustering error is low for subset sizes of the order of the number of subspaces. As demonstrated in Figure 7 (b), to achieve a clustering error of less than 0.01, Lasso requires times more optimizations than OWL. The plots suggest that the OWL approach could potentially achieve a speedup of up to .

Figure 9: Clustering error for varying in Algorithm LABEL:owlalgo versus varying noise levels, , for Lasso (left) and OWL (right) minimization based SSC. Affinity matrix for each method is generated by running only a random subset of total optimizations.

Effect of affinity and number of points sampled. We vary the amount of correlation between subspaces and the number of points sampled from each subspace and study its effect on the clustering error for different values of in Figure 8. The subspaces with varying affinity are generated using the method described in B2 and subspaces for varying are generated using B1. The affinity is varied in the range where and we vary the sampling density . Recall that the affinity is low when the subspaces are nearly orthogonal and high when the subspaces overlap significantly (it is equal to one when one subspace is contained in the other). The tuning parameters are fixed throughout the experiment. As expected, the clustering error increases for higher values of affinity and OWL () provides no benefit over Lasso (). On the other hand, OWL produces small values of error for most values of .

2 Mean 1.52 16.92 18.44 3.62 1.39
Median 0.00 12.77 16.92 0.00 0.00
3 Mean 4.40 27.96 28.58 8.28 5.21
Median 0.56 30.98 29.67 2.76 1.02
Table 2: Clustering error (%) of different algorithms on the Hopkins 155 dataset.
Motions (L) SSC OSC
2 1.83 1.49
3 4.40 5.16
All 2.41 2.32
Table 3: Mean clustering error (%) on the Hopkins 155 dataset with the 4-dimensional data points obtained by applying PCA.
Figure 10: (a) Mean clustering error for Hopkins dataset. (b) Mean clustering error for MNIST dataset.

Effect of noise. To demonstrate the performance of the algorithms under noise, we perturb each unit norm data point by a noisy vector chosen uniformly at random from the sphere of radius and normalize again to have unit norm. The noisy model has been studied in [20] using the Lasso version of SSC. Figure 9 shows the empirical average of clustering errors obtained by the three methods across different subspaces generated using B1 by varying the noise level and the number of optimizations run. The OWL method with achieves low clustering error for noise levels as high as providing a speedup of up to .

5 Experiments with Real Data

We compare our algorithm with the existing ones in the applications of motion segmentation and clustering handwritten digits.

For motion segmentation, we used Hopkins155 dataset [16]. It contains 155 video sequences of 2 or 3 motions. Table 2 summarizes clustering errors of different algorithms on the Hopkins155 dataset. For ease of comparison, the error values for TSC, SSC-OMP and NSN are populated from experiments conducted in [12] where the authors optimized the parameters for the existing algorithms. The parameters for SSC were set as provided in the source code. The OWL parameters are set as follows: for all real data experiments we use the same value as SSC, we set (rounded off to nearest integer) and set such that or , whichever, if any, leads to a non-trivial solution. Figure 10(a) shows the effect of running a subset of optimizations selected at random from the set of all optimizations for . OSC outperforms or performs about the same as SSC in most cases with comparable running times. The SSC and OSC parameters in this application lead to relatively dense affinity matrices in both cases. This results in low clustering errors with fewer optimizations supporting the theory. Table 3 summarizes results of applying the SSC and OSC algorithms to the dataset projected into a 4-dimensional subspace using PCA suggesting further speedups without loss in performance.

We also use the MNIST test data set [11] that contains 10,000 centered 28 28 pixel images of handwritten digits, i.e.,

it contains many more points per subspace. The empirical mean and standard deviation of the CE are computed by averaging over 100 of the following problem instances. We choose the digits {2, 4, 8} and we choose

optimizations uniformly at random from the set of all optimizations for . Here we use the default parameters for SSC and set OWL parameters in similar fashion to the motion segmentation experiments. The results are summarized in Figure 10(b).

6 Conclusion

In this paper we have proposed a new approach to subspace clustering based on Ordered Weighted (OWL) minimization which performs simultaneous regression and clustering of correlated variables. The clustering property of the OWL, combined with ideas from random geometric graph theory, allows us to prove that OSC tends to select more points from the correct subspaces in each regression compared to SSC. In the ideal case, where subspaces are orthogonal and the number of points per subspace is sufficiently large, then OSC can succeed with just optimizations (gain factor of roughly ). This key feature of OSC makes accurate clustering possible based on regression solutions for only a small subset of the total dataset, significantly reducing the computational complexity compared to SSC.


The authors thank Ulaz Ayaz for fruitful discussions.


  • [1] M. Bogdan, E. van den Berg, C. Sabatti, W. Su, and E. J. Candès. Slope?adaptive variable selection via convex optimization. The annals of applied statistics, 9(3):1103, 2015.
  • [2] H. D. Bondell and B. J. Reich. Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with oscar. Biometrics, 64(1):115–123, 2008.
  • [3] J. Dall and M. Christensen. Random geometric graphs. Physical review E, 66(1):016121, 2002.
  • [4] E. L. Dyer, A. C. Sankaranarayanan, and R. G. Baraniuk.

    Greedy feature selection for subspace clustering.

    The Journal of Machine Learning Research

    , 14(1):2487–2517, 2013.
  • [5] E. Elhamifar and R. Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence, 35(11):2765–2781, 2013.
  • [6] B. Eriksson, L. Balzano, and R. Nowak. High-rank matrix completion. In Artificial Intelligence and Statistics, pages 373–381, 2012.
  • [7] M. Figueiredo and R. Nowak. Ordered weighted l1 regularized regression with strongly correlated covariates: Theoretical aspects. In Artificial Intelligence and Statistics, pages 930–938, 2016.
  • [8] R. Heckel and H. Bölcskei. Robust subspace clustering via thresholding. IEEE Transactions on Information Theory, 61(11):6320–6342, 2015.
  • [9] R. Heckel, M. Tschannen, and H. Bölcskei. Dimensionality-reduced subspace clustering. Information and Inference: A Journal of the IMA, page iaw021, 2017.
  • [10] J. Ho, M.-H. Yang, J. Lim, K.-C. Lee, and D. Kriegman. Clustering appearances of objects under varying illumination conditions. In

    Computer vision and pattern recognition, 2003. Proceedings. 2003 IEEE computer society conference on

    , volume 1, pages I–I. IEEE, 2003.
  • [11] Y. LeCun, C. Cortes, and C. J. Burges.

    The mnist database. Accessed: 2018.
  • [12] D. Park, C. Caramanis, and S. Sanghavi. Greedy subspace clustering. In Advances in Neural Information Processing Systems, pages 2753–2761, 2014.
  • [13] M. Penrose. Random geometric graphs. Number 5. Oxford University Press, 2003.
  • [14] M. Soltanolkotabi, E. J. Candes, et al.

    A geometric analysis of subspace clustering with outliers.

    The Annals of Statistics, 40(4):2195–2238, 2012.
  • [15] M. Soltanolkotabi, E. Elhamifar, E. J. Candes, et al. Robust subspace clustering. The Annals of Statistics, 42(2):669–699, 2014.
  • [16] R. Tron and R. Vidal. A benchmark for the comparison of 3-d motion segmentation algorithms. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pages 1–8. IEEE, 2007.
  • [17] R. Vidal. Subspace clustering. IEEE Signal Processing Magazine, 28(2):52–68, 2011.
  • [18] R. Vidal, R. Tron, and R. Hartley. Multiframe motion segmentation with missing data using powerfactorization and gpca. International Journal of Computer Vision, 79(1):85–105, 2008.
  • [19] Y. Wang, Y.-X. Wang, and A. Singh. A deterministic analysis of noisy sparse subspace clustering for dimensionality-reduced data. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1422–1431, 2015.
  • [20] Y.-X. Wang and H. Xu. Noisy sparse subspace clustering. Journal of Machine Learning Research, 17(12):1–41, 2016.
  • [21] A. Y. Yang, J. Wright, Y. Ma, and S. S. Sastry. Unsupervised segmentation of natural images via lossy data compression. Computer Vision and Image Understanding, 110(2):212–225, 2008.
  • [22] C. You, D. Robinson, and R. Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3918–3927, 2016.
  • [23] X. Zeng and M. A. Figueiredo. The ordered weighted norm: Atomic formulation, projections, and algorithms. arXiv preprint arXiv:1409.4271, 2014.


Appendix A Properties of OWL

Subgradient condition for OWL:

When defined as above, we can write the subdifferential vector as

Appendix B Proof of Lemma 2

Proof 1

Let . Note that .

From optimality conditions, we have .

Consider a perturbation .
Let and be permutation matrices such that and . Notice that these permutation matrices may not be unique.

For sufficiently small such that and the group ordering doesn’t change i.e, there exist and such that ,

where the last inequality follows since and . The weights on will only decrease. Recall from optimality conditions we have , this gives us

Let and if . Consider the last term,

from assumption .

This implies and the claim follows.

Appendix C Proof of Lemma 3

Proof 2


Since minimizes the first optimization and , we have the following

Let , then

Let and be permutation matrices such that:

Note that by definition of OWL norm . Let be the support of , then

Plugging into the inequality above gives

since is optimal there exists such that

using this

Thus we have