Efficient Solvers for Sparse Subspace Clustering
share
Sparse subspace clustering (SSC) is a popular method in machine learning and computer vision for clustering n data points that lie near a union of low-dimensional linear or affine subspaces. The standard models introduced by Elhamifar and Vidal express each data point as a sparse linear or affine combination of the other data points, using either ℓ_1 or ℓ_0 regularization to enforce sparsity. The ℓ_1 model, which is convex and has theoretical guarantees but requires O(n^2) storage, is typically solved by the alternating direction method of multipliers (ADMM) which takes O(n^3) flops. The ℓ_0 model, which is preferred for large n since it only needs memory linear in n, is typically solved via orthogonal matching pursuit (OMP) and cannot handle the case of affine subspaces. Our first contribution is to show how ADMM can be modified using the matrix-inversion lemma to take O(n^2) flops instead of O(n^3). Then, our main contribution is to show that a proximal gradient framework can solve SSC, covering both ℓ_1 and ℓ_0 models, and both linear and affine constraints. For both ℓ_1 and ℓ_0, the proximity operator with affine constraints is non-trivial, so we derive efficient proximity operators. In the ℓ_1 case, our method takes just O(n^2) flops, while in the ℓ_0 case, the memory is linear in n. This is the first algorithm to solve the ℓ_0 problem in conjunction with affine constraints. Numerical experiments on synthetic and real data demonstrate that the proximal gradient based solvers are state-of-the-art, but more importantly, we argue that they are more convenient to use than ADMM-based solvers because ADMM solvers are highly sensitive to a solver parameter that may be data set dependent.
READ FULL TEXT