A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries

1 Introduction

The dictionary learning problem is as follows: given observations $Y$ , the task is to factorize it as

Y = A X, Y \in R^{d \times n}, A \in R^{d \times r}, X \in R^{r \times n},

(1)

where $X$ is referred to as the coefficient matrix and the columns of $A$ are referred to as the dictionary elements. There are indeed infinite factorizations for (1) unless further constraints are imposed. A natural assumption is that the coefficient matrix $X$ is sparse, and in fact, that each sample $y_{i}$ selects a sparse subset of dictionary elements from $A$ . This instance of dictionary learning is popularly known as the sparse coding problem [30, 25]. It has been argued that sparse coding can provide a succinct representation of the observed data, given only unlabeled samples [25]

. Through this lens of unsupervised learning, dictionary learning has received an increased attention from the machine learning community in the last few years; see Section

1.2 for a brief survey.

Although the above problem has been extensively studied, most of the methods are heuristic and lack guarantees. Spielman et. al

[31] provide exact recovery results for this problem, when the coefficient matrix has Bernoulli-Gaussian entries and the dictionary matrix

A \in R^{r \times d}

has full column rank. This condition entails that the dictionary is undercomplete, i.e., the observed dimensionality needs to be greater than the number of dictionary elements

(r \leq d)

. However, for most practical settings, it has been argued that overcomplete representations, where

r ≫ d

, are far more relevant, and can provide greater flexibility in modeling as well as greater robustness to noise [26, 12]. Moreover, in the context of blind source separation (BSS) of audio, image or video signals, the dictionary learning problem is typically overcomplete, since there are more sources than observations [15]. In this work, we provide guaranteed methods for learning overcomplete dictionaries.

1.1 Summary of Results

In this paper we present a novel algorithm for the estimation of overcomplete dictionaries. The algorithm can be seen as a clustering style method

followed by a singular value decomposition (SVD) within each cluster resulting in an estimate for each dictionary element. The clusters are formed based on the magnitudes of the correlation between pairs of samples. Under our probabilistic model of generating data as well as assumptions on the coefficients and dictionaries, it can be guaranteed that such a procedure approximately recovers the unknown overcomplete dictionary. Under further conditions, it is often possible to start with this approximate solution and perform additional post-processing on it to obtain arbitrarily good estimates of the dictionary. We present one such set of conditions under which

sparse regression can be used for this post-processing. More advanced post-processing methods have been developed in subsequent works [1, 8].

We consider a random coefficient matrix, where each column of $X$ has $s$ non-zero entries which are randomly chosen, i.e., each sample $y_{i}$ selects $s$ dictionary elements uniformly at random. We additionally assume that the dictionary elements are pairwise incoherent and that the dictionary matrix satisfies a certain bound on the spectral norm. Under these conditions, we establish that our algorithm estimates the dictionary elements with bounded (constant) error when the number of samples scales as $n = O (r (log r + log d))$ , and when the sparsity $s = O (d^{1 / 4}, r^{1 / 4})$ . To the best of our knowledge, this is the first result of its kind which analyzes the global recovery properties of a computationally efficient procedure in the setup of overcomplete dictionary learning.

In the special case when the coefficients are ${- 1, 0, 1}$ -valued with zero mean, the resulting solution from the first step can be further plugged into any sparse regression algorithm for estimating the coefficients given this dictionary estimate. Under a more stringent sparsity constraint: $s = O (d^{1 / 5}, r^{1 / 6})$ , it can be shown that this second step will recover the coefficients exactly even from this approximate dictionary, which then also leads to an exact recovery of the dictionary by solving the linear system. Hence, we provide a simple method for exactly recovering the unknown dictionary in this special case. A natural generalization of this procedure to general weights is analyzed using alternating minimization procedure in a subsequent work [1].

We outline our method as well as our analysis techniques in Section 1.3. This is the first work to provide a tractable method for guaranteed recovery of overcomplete dictionaries, and we discuss the previous results below. Finally, concurrently with our work, an approximate recovery result with a similar procedure was recently announced by Arora et al. [8]. A detailed discussion comparing our and their results is presented in Section 1.2.

1.2 Related Works

This work overlaps with and relates to prior works in many different communities and we discuss them below in turn.

Dictionary Learning:

Hillar and Sommer [20]

consider conditions for identifiability of sparse coding and establish that when the dictionary succeeds in reconstructing a certain set of sparse vectors, there exists a unique sparse coding, up to permutation and scaling. However, the number of samples required to establish identifiability is exponential in

r

for the general case. In contrast, we show that efficient recovery is possible using

O (r (log r + log d))

samples, albeit under additional conditions such as incoherence among the dictionary elements.

Spielman et. al [31] provide exact recovery results for a $ℓ_{1}$ based method in the undercomplete setting, where $r \leq d$ . In contrast, we allow for the overcomplete setting where $r > d$ . There exist a plethora of heuristics for dictionary learning, which work well in practice in many contexts, but lack theoretical guarantees. For instance, Lee et. al. propose an iterative $ℓ_{1}$ and $ℓ_{2}$ optimization procedures [25]. This is similar to the the method of optimal directions (MOD) proposed in [16]. Another popular method is the so-called K-SVD, which iterates between estimation of $X$ and given an estimate of $X$ , updates the dictionary estimate using a spectral procedure on the residual. Other works consider more sophisticated methods from an optimization viewpoint while still alternating between dictionary and coefficient updates [24, 18]. Geng et al. [18] and Jenatton et al. [23] study the local optimality properties of an alternating minimization procedure. In contrast, our work focuses on global properties of a more combinatorial procedure than several of the above works which are more optimization flavored. The upshot is that our procedure, while still being computationally quite efficient, is able to guarantee global bounds on the quality of the solution obtained.

Recent works [34, 29, 27, 32] provide generalization bounds and algorithmic stability for predictive sparse coding, where the goal of the learned sparse representation is to obtain good performance on some predictive task. This differs from our framework since we do not consider predictive tasks here, but the accuracy in recovering the underlying dictionary elements.

Finally, our results are closely related to the very recent work of Arora et al. [8], carried out independently and concurrently with our work. There are however some important distinctions: we require only $O (r)$ samples in our analysis, while Arora et al. [8] require $O (r^{2})$ samples in their result. At the same time, their analysis yields milder conditions on the sparsity level $s$ in terms of its dependence on $r$ and $d$ . Following this work, Arora et al. [8] and Agarwal et al. [1] also developed a post-processing techniques which can be thought of as a more advanced variant of the simpler sparse-regression step that we analyze. These subsequent works view the methods developed here as initialization procedures to alternating optimization schemes.

Blind Source Separation/ICA/Topic Models:

The problem of dictionary learning is applicable to blind source separation (BSS), where the rows of $X$ are signals from the sources and $A$ represents the linear mixing matrix. The term blind implies that the dictionary matrix $A$ is unknown and needs to be jointly estimated with the coefficient matrix $X$ , given samples $Y$

. This problem has been extensively studied and the most popular setting is the independent component analysis (ICA), where the sources are assumed to be independent. In contrast, for the sparse component analysis problem, no assumptions are made on the statistics of the sources. Many works provide guarantees for ICA in the undercomplete setting, where there are fewer sources than observations

[21, 9, 4] and some works provide guarantees in the overcomplete setting [14, 19]. However, the techniques are very different since they rely on the independence among the sources. The problem of learning topic models can be cast as a similar factorization problem, where

A

now corresponds to the topic-word matrix and

X

corresponds to the proportions of topics in various documents. There are various recent works providing guaranteed methods for learning topic models, e.g [2, 7, 6, 5]. However, these works make different assumptions on either

A

X

or both to guarantee recovery. For instance, the work [7] assumes that the topic-word matrix

A

has rows such that for each column, only the entry corresponding to that column is non-zero. The work [6] assumes expansion conditions on

A

and provides recovery through

ℓ_{1}

-based optimization. We note that the techniques of [6] are related to those employed by Spielman et. al [31] for dictionary learning, but make different assumptions. All these works only deal with the undercomplete setting. The recent work [5] considers topic models in the overcomplete setting, and provides guarantees when

A

satisfies certain higher order expansion conditions. The techniques are very different from the ones employed here since they involve higher order moments and tensor forms.

Connection to Learning Overlapping Communities:

Our initial step for estimating the dictionary elements involves finding large cliques in the sample correlation graph, where the nodes are the samples and the edges represent sufficiently large correlations among the endpoints. The clique finding problem is a special instance of the overlapping community detection problem, which has been studied in various contexts, e.g. [3, 11, 10, 22, 28]. However, the correlation graph here has different kinds of constraints than the ones studied before as follows. In our setting involving noise-free dictionary learning, each community corresponds to a clique and there are no edges across two different communities. In contrast, many works on community detection are concerned about handling noise efficiently, where each community is not a full clique, and there are edges across different communities. Here, we need to learn overlapping communities, while many community detection methods limit to learning non-overlapping ones. In our setting, we argue that the overlap across different communities is small under a random coefficient matrix, and thus, we can find the communities efficiently through simple random sampling and neighborhood testing procedures.

1.3 Overview of Techniques

As stated earlier, our main algorithm consists of a clustering procedure which yields an approximate estimate of the dictionary. This estimate can be subsequently post-processed for exact recovery of the dictionary under certain further conditions. Below we give the outline and the main intuition underlying these procedures and their analysis.

Dictionary estimation via clustering:

This step first involves construction of the sample correlation graph $G_{corr (ρ)}$ , where the nodes are samples ${y_{1}, y_{2}, \dots y_{n}}$ and an edge $(y_{i}, y_{j}) \in G_{corr (ρ)}$ implies that $| ⟨ y_{i}, y_{j} ⟩ | > ρ$ , for some $ρ > 0$ . We then employ a clustering procedure on the graph to obtain a subset of samples, which are then employed to estimate each dictionary element. Roughly, we search for large cliques in the correlation graph and obtain a spectral estimate of each dictionary element using samples from such sets.

Key intuitions for the clustering procedure:

The core intuitions can be described in terms of the relationships between the two graphs, viz., the coefficient bipartite graph $B_{coeff}$ and the sample correlation graph $G_{corr}$ , shown in Figures (a)a and (b)b. As described earlier, the correlation graph $G_{corr}$ consists of edges between well correlated samples. The coefficient bipartite graph $B_{coeff}$ consists of dictionary elements ${a_{i}}$ on one side and the samples ${y_{i}}$ on the other, and the bipartite graph $B_{coeff}$ encodes the sparsity pattern of the coefficient matrix $X$ . In other words, it maps the dictionary elements ${a_{i}}$ to samples ${y_{i}}$ on which they are supported on and $N_{B} (y_{i})$ denotes the neighborhood of $y_{i}$ in the graph $B_{coeff}$ .

Now given this bipartite graph $B_{coeff}$ , for each dictionary element $a_{i}$ , consider a set of samples²²2Note that such a set need not be unique. which (pairwise) have only one dictionary element $a_{i}$ in common, and denote such a set by $C_{i}$ i.e.

C_{i} := {y_{k}, k \in S : N_{B} (y_{k}) \cap N_{B} (y_{l}) = a_{i}, \forall k, l \in S} .

(2)

For a random coefficient matrix (resulting in a random bipartite graph), we argue that there exists (large) sets $C_{i}$ , for each $i \in [r]$ , which consists of a large fraction of $N_{B} (a_{i})$ , and no two elements $a_{i}$ and $a_{j}$ have a large fraction of samples in common. In other words, for random coefficient matrices, we see a diversity in the dictionary elements among the samples, and this can be viewed as an expansion property from the dictionary elements to the set of samples. We exploit this property to establish success for our method.

Our subsequent analysis is broadly divided into two parts, viz., establishing that (large) sets ${C_{i}}$ can be found efficiently, and that the dictionary elements can be estimated accurately once such sets ${C_{i}}$ are found. We establish that the sets ${C_{i}}$ are cliques in the correlation graph when the dictionary elements are incoherent, as shown in Figure (b)b. Combined with the previous argument that the different sets $C_{i}$ ’s have only a small amount of overlap for random coefficient matrices, we argue that these sets can be found efficiently through simple random sampling and neighborhood testing on the correlation graph. Once a large enough set $C_{i}$ is found, we argue that under incoherence, the dictionary element $a_{i}$ can be estimated accurately through SVD over the samples in $C_{i}$ .

Sparse regression for post-processing:

This is a relatively straightforward procedure. Once an initial estimate of the dictionary matrix is obtained, we estimate the coefficient matrix $X$ through any sparse regression procedure (such as Lasso) and then perform thresholding on the recovered coefficients. Now, we re-estimate the dictionary, given this coefficient matrix, by solving another linear system. This provides us with a final estimate of both the dictionary as well as the coefficient matrix.

Since we only have a noisy estimate of the dictionary, our analysis here is slightly different from the usual analysis for a sparse linear system. The noise in our system is dependent on the approximate dictionary employed, which differs from the typical statistical setting, where noise is assumed to be independent. We exploit the known guarantees available for Lasso under deterministic noise [13] for our setting. Combining Lasso with a simple thresholding procedure, we guarantee exact recovery of the coefficient matrix, albeit under a more stringent condition on the sparsity and the coefficient values (namely zero mean and ${- 1, 0, 1}$ -valued ). The dictionary is then re-estimated by solving another linear system, which is of course correct owing to the exact estimation of the coefficient matrix.

k[c]

r

d[c]

n

h1[c]

a_{1}

h2[c]

a_{2}

h3[c]

a_{r}

x1[c]

y_{1}

x2[c]

y_{2}

x4[c] x5[c] x6[c]

y_{n}

A[c]

X

C1[c]

C_{1}

Cr[l]

C_{r}

C12[l]

C_{12}

(a) a

Ci[l]

C_{i}

Ci[l]

C_{i}

Cj[c]

C_{j}

Cj[l]

C_{j}

(b) b

Figure 3: Coefficient bipartite graph and the sample correlation graph.

2 Method and Guarantees

Notation:

Let $[n] := {1, 2, \dots, n}$ and for a vector $w$ , let $Supp (w)$ denote the support of $w$ , i.e. the set of indices where $w$ is non-zero. Let $∥ w ∥$ denote the $ℓ_{2}$ norm of vector $w$ , and similarly for a matrix $W$ , $∥ W ∥$ denotes its spectral norm. Let $A = [a_{1} | a_{2} | \dots | a_{r}]$ , where $a_{i}$ denotes the $i\tiny th$ column, and similarly for $Y = [y_{1} | y_{2} | \dots y_{n}]$ and $X = [x_{1} | \dots | x_{n}]$ . For a graph $G = (V, E)$ , let $N_{G} (i)$ denote set of neighbors for node $i$ in $G$ .

2.1 Clustering procedure and its analysis

We start with presenting the main algorithm of our work and bound the recovery error under certain assumptions.

2.1.1 Algorithm

Our main algorithm is presented in Algorithm 1. Given samples $Y$ , we first construct the correlation graph $G_{corr (ρ)}$ , where the nodes are samples ${y_{1}, y_{2}, \dots y_{n}}$ and an edge $(y_{i}, y_{j}) \in G_{corr (ρ)}$ implies that $| ⟨ y_{i}, y_{j} ⟩ | > ρ$ , for some threshold $ρ > 0$ . We then determine a good subset of samples via a clustering procedure on the graph as follows: we first randomly sample an edge $(y_{i^{*}}, y_{j^{*}}) \in G_{corr (ρ)}$ and then consider the intersection of their neighborhoods, denoted by $ˆ S$ . We then employ UniqueIntersection routine in Procedure 1 to determine if $ˆ S$ is a “good set” for estimating a dictionary element, and this is done by ensuring that the set $ˆ S$ has sufficient number of mutual neighbors³³3For convenience to avoid dependency issues, in Procedure 1, we partition $ˆ S$ into sets consisting of node pairs and determine if there are sufficient number of node pairs which are neighbors. in the correlation graph. Once $ˆ S$ is determined to be a good set, we then proceed by estimating the matrix $ˆ M$ using samples in $ˆ S$

and output its top singular vector as the estimate of a dictionary element. The method is repeated over all edges in the correlation graph to ensure that all the dictionary elements get estimated with high probability.

0: Samples

Y = [y_{1} | \dots | y_{n}]

. Correlation threshold

ρ

. Desired separation parameter

ϵ

between recovered dictionary elements.

0: Initial Dictionary Estimate

¯ A

Construct correlation graph

G_{corr (ρ)}

s.t.

(y_{i}, y_{j}) \in G_{corr (ρ)}

when

| ⟨ y_{i}, y_{j} ⟩ | > ρ

Set

¯ A \leftarrow \emptyset

for each edge

(y_{i^{*}}, y_{j^{*}}) \in G_{corr (ρ)}

ˆ S \leftarrow N_{G_{corr (ρ)}} (y_{i^{*}}) \cap N_{G_{corr (ρ)}} (y_{j^{*}})

if UniqueIntersection

(ˆ S, G_{corr (ρ)})

then

ˆ L \leftarrow \sum_{y \in ˆ S} y y^{⊤}

and

¯ a \leftarrow u_{1}

, where

u_{1}

is top singular vector of

ˆ L

{min}_{b \in ¯ A} ∥ ¯ a - b ∥ > 2 ϵ_{dict}

then

¯ A \leftarrow ¯ A \cup ¯ a

end if

end for

Return

¯ A

Algorithm 1 DictionaryLearn

(Y, ϵ_{dict}, ρ)

: Clustering approach for estimating dictionary elements.

0: Set

S

with

2 ¯ ¯ ¯ n

vectors

y_{1}, \dots y_{2 ¯ ¯ ¯ n}

and graph

G

with

y_{1}, \dots, y_{2 ¯ ¯ ¯ n}

as nodes.

0: Indicator variable UNIQUE_INT

Partition

S

into sets

S_{1}, \dots, S_{¯ ¯ ¯ n}

such that each

| S_{t} | = 2

| {t | S_{t} \in G} | > \frac{61 ¯ ¯ ¯ n}{64}

then

UNIQUE_INT

\leftarrow 1

else

UNIQUE_INT

\leftarrow 0

end if

Return UNIQUE_INT

Procedure 1 UniqueIntersection

(S, G)

: Determine if samples in

S

have a unique intersection.

2.1.2 Assumptions and Main Result

Assumptions:

We now provide guarantees for the proposed method under the following assumptions on $A$ and $X$ .

Unit-norm Dictionary Elements: All the elements are normalized: $∥ a_{i} ∥ = 1$ , for $i \in [r]$ .
Incoherent Dictionary Elements: We assume pairwise incoherence condition on the dictionary elements, for some constant $μ_{0} > 0$ ,

$| ⟨ a_{i}, a_{j} ⟩ | < \frac{μ_{0}}{\sqrt{d}} .$ (3)
Spectral Condition on Dictionary Elements: The dictionary matrix has bounded spectral norm, for some constant $μ_{1} > 0$ ,

$∥ A ∥ < μ_{1} \sqrt{\frac{r}{d}} .$ (4)
Entries in Coefficient Matrix: We assume that the non-zero entries of $X$ are drawn from a zero-mean distribution supported on $[- M, - m] \cup [m, M]$ for some fixed constants $m$ and $M$ .
Sparse Coefficient Matrix: The columns of coefficient matrix have bounded number of non-zero entries $s$ which are selected randomly, i.e.

$| Supp (x_{i}) | = s, \forall i \in [n] .$ (5)

We require $s$ to be

$s < c min (\sqrt{\frac{m^{2} \sqrt{d}}{2 M^{2} μ_{0}}}, \sqrt[3]{\frac{r}{1536}}),$ (6)

for some small enough constant $c$ .
Sample Complexity: Given a parameter $α \in (0, 1 / 20)$ (which is related to the error in recovery of dictionary, see Theorem 2.1), and a universal constant $c > 0$ , choose $δ > 0$ and the number of samples $n$ such that

$n := n (d, r, s, δ, α) = \frac{c r}{α^{2} s} log \frac{d}{δ}, n^{2} δ < 1.$
Choice of Threshold for Correlation Graph: The correlation graph $G_{corr (ρ)}$ is constructed using threshold $ρ$ such that

$ρ = \frac{m^{2}}{2} - \frac{s^{2} M^{2} μ_{0}}{\sqrt{d}} > 0.$ (7)
Choice of Separation Parameter $ϵ_{dict}$ between Estimated Dictionary Elements: This is the desired accuracy of the estimated dictionary elements to the true dictionary elements using just the initialization step. It can be chosen to be:

$\frac{32 s M^{2}}{m^{2}} (\frac{μ_{1}}{\sqrt{d s}} + \frac{μ_{1}^{2}}{d} + \frac{s^{3}}{r} + α^{2} + \frac{α}{\sqrt{s}}) < ϵ_{dict}^{2} < \frac{1}{4} .$ (8)

The assumption $(A 1)$ on normalization is without loss of generality since we can always rescale the dictionary elements and the corresponding coefficients and obtain the same observations. The assumption $(A 2)$ on incoherence is crucial to our analysis. In particular, incoherence also leads to a bound on the RIP constant; see Lemma A.5 in Appendix A.6. The assumption $(A 3)$ provides a bound on the spectral norm of $A$ .

The assumption $(A 4)$ assumes that the non-zero entries of $X$ are drawn from a zero-mean distribution with natural upper and lower bounds on the coefficients. Note that a similar assumption is made in the work of Arora et.al [8].

The assumption $(A 5)$ on sparsity in the coefficient matrix is crucial for identifiability of dictionary learning problem. We require for the sparsity to be not too large for recovery.

The assumption $(A 6)$ provides a bound on sample complexity. We subsequently establish that in order to have decaying error for recovery of dictionary elements, we require $n = ω (r)$ samples for recovery. Thus, we obtain a nearly linear sample complexity for our method.

Assumption $(A 7)$ specifies the threshold for the construction of the correlation graph. Intuitively, we require a threshold such that we can distinguish pairs of samples which share a dictionary element from those which do not.

Main Result:

We now present our main result which bounds the error in the estimates of Algorithm 1.

Theorem 2.1 (Approximate recovery of dictionary).

Suppose the output of Algorithm 1 is $¯ A$ . Then with probability greater than $1 - 2 n^{2} δ$ , there exists a permutation matrix $P$ such that:

ϵ_{A}^{2} := min i \in [r] min z \in {- 1, + 1} {∥ ∥ z a_{i} - (P ¯ A)_{i} ∥ ∥}_{2}^{2} < \frac{32 s M^{2}}{m^{2}} (\frac{μ_{1}}{\sqrt{d s}} + \frac{μ_{1}^{2}}{d} + \frac{s^{3}}{r} + α^{2} + \frac{α}{\sqrt{s}}) .

(9)

Remark:

Note that we have a sign ambiguity in recovery of the dictionary elements, since we can exchange the signs of the dictionary elements and the coefficients to obtain the same observations. The assumption on sparsity in $(A 4)$ implies that the first two terms in (9) decay. For the third term in (9) to decay, we require $s = o (r^{1 / 4})$ instead of $s = O (r^{1 / 3})$ as in $(A 4)$ . Moreover, we require that $α^{2} s = o (1)$ . Since the sample complexity in $(A 7)$ scales as $n = Ω (\frac{r}{α^{2} s})$ , we require $n = ω (r)$ samples for recovery of dictionary with decaying error. Thus, we obtain a near linear sample complexity for our method. We observe that the error in our estimation depends inversely on dimension-related quantities such as $d$ and $r$ and not on the number of samples $n$ . This is because the errors in our estimates arise from errors in SVD step, specifically from the discrepancy between the SVD vector and the dictionary element responsible for a cluster. Even the population SVD will suffer from an approximation error here, which is responsible for our error bound, but the probability in the error bound improves with the number of samples as we get closer and closer to the population SVD estimate.

2.2 Post-processing for binary coefficients

We now present the post-processing step which will be analyzed under a more stringent condition on the coefficients.

2.2.1 Algorithm

0: Samples

Y

, approximate dictionary

¯ A

and accuracy parameter

ϵ_{coeff}

sign (X)

returns a matrix with signs of the entries of

X

1: for samples

i = 1, 2, \dots, n

2: Estimate

{ˆ x}_{i} = arg min x \in R^{r} ∥ x ∥_{1}, % subject to ∥ y_{i} - ¯ A x ∥_{2} \leq ϵ_{coeff} .

(10)

3: Threshold:

ˆ X \leftarrow sign (ˆ X)

4: end for

5: Estimate

ˆ A = Y {ˆ X}^{T} (ˆ X {ˆ X}^{T})^{- 1}

6: Normalize:

{ˆ a}_{i} = \frac{{ˆ a}_{i}}{∥ {ˆ a}_{i} ∥_{2}}

ˆ A

Algorithm 2 RecoverCoeff

(Y, ¯ A, ϵ_{coeff})

: Exact Recovery through lasso

Once we obtain an estimate of the dictionary elements, we proceed to estimate the coefficient matrix. The main observation at this step is that the coefficient vector $x_{i}$ for each sample $y_{i}$ is a $s$ -sparse vector in $r$ -dimensions. Hence, recovering the coefficients would be a standard sparse linear problem if we knew the dictionary $A$ exactly. Our analysis will show that even an approximately correct dictionary $¯ A$ from Algorithm 1 suffices to provide guarantees for this recovery. Once the coefficients are estimated, the dictionary can be re-estimated by solving another linear system. The procedure is formally described in Algorithm 2. We do not prescribe any particular choice of computational procedure to solve the optimization problem (10), but there are many algorithms available in standard literature. As a concrete example, the GraDeS algorithm of Garg and Khandekar [17] or OMP of Tropp and Gilbert [33]works in our setting.

2.2.2 Exact recovery for bernoulli coefficients

Our second result is that under stronger conditions than before, it is possible to exactly recover the unknown dictionary $A$ with high probability. This result will be obtained by initializing Algorithm 2 with the output of Algorithm 1. We start with the additional assumptions, putting restrictions on the allowed sparsity level $s$ as a function of $r$ and $d$ .

Assumption B1 (Conditions for exact recovery).

The non-zero coefficients in coefficient matrix $X$ are zero-mean Bernoulli ${- 1, 1}$ . This corresponds to setting $M = m = 1$ in Assumption (A4).

The sparsity level $s$ , and the number of dictionary elements $r$ and the observed dimension $d$ satisfy

32 s (\frac{μ_{1}}{\sqrt{d s}} + \frac{μ_{1}^{2}}{d}) \leq \frac{1}{1200 s^{2}}, and \frac{32 s^{4}}{r} \leq \frac{1}{1200 s^{2}} .

The constant $α$ in Theorem 2.1 satisfies

32 s (α^{2} + \frac{α}{\sqrt{s}})

\leq \frac{1}{1200 s^{2}} .

The number of samples $n$ , in addition to assumption $(A 6)$ , satisfies

n \geq \frac{4 r}{c_{0}} log \frac{d}{δ},

where $c_{0}$ is a universal constant.

The accuracy parameter $ϵ_{coeff}$ in Algorithm 2 is chosen as $ϵ_{coeff} = s ϵ_{A}$ , where $ϵ_{A}$ is the error in estimating the dictionary elements in (9).

Theorem 2.2 (Exact recovery for bernoulli coefficients).

Under the conditions of Theorem 2.1, and suppose, in addition Assumption B1 holds, then the output $ˆ A$ of Algorithm 2 initialized with Algorithm 1 satisfies $ˆ A = A$ up to permutation of columns, with probability at least $1 - 3 n^{2} δ$ .

Remark:

Assumption B1 for exact recovery places more stringent conditions on the distribution of the coefficients and the sparsity level $s$ , compared to $(A 4)$ for approximate recovery. While for approximate recovery, we require $s = O (d^{1 / 4}, r^{1 / 4})$ , in Assumption B1, we require $s = O (d^{1 / 5}, r^{1 / 6})$ for exact recovery. Note that the additional constraint on sample complexity $n$ in Assumption B1 still has the same scaling, and thus, $n = O (r (log r + log d))$ suffices both for approximate and exact recovery.

We also observe that the result of Theorem 2.2 relies on Algorithm 1 as the initialization procedure, but in principle we can also use a different approximate recovery procedure to initialize Algorithm 2. In particular, a different initialization procedure with a better error guarantee would also directly translate to better recovery properties in the second step, in terms of the assumptions relating $s$ to $r$ and $d$ . Understanding these issues appears to be an interesting direction for future research.

3 Proofs of main results

In this section we will present the proofs of our main results, Theorems 2.1 and 2.2. We will start by presenting a host of useful lemmas, and sketch out how they fit together to yield the main results before moving on to the proofs.

3.1 Correlation graph properties

In this section we will present some useful properties of the correlation graph $G_{corr (ρ)}$ described in Section 1.3. Recall that $G_{corr (ρ)}$ , where the nodes are samples ${y_{1}, y_{2}, \dots y_{n}}$ and an edge $(y_{i}, y_{j}) \in G_{corr (ρ)}$ implies that $| ⟨ y_{i}, y_{j} ⟩ | > ρ$ , for some $ρ > 0$ . This is employed by Algorithm 1 as a proxy for identifying samples which have common dictionary elements. We now make this connection concrete in the next few lemmas. For this we also recall our notation $N_{B} (y)$ which is the neighborhood of a sample $y$ in the coefficient bipartite graph (see Figure (a)a), that is, the set of dictionary elements that combine to yield $y$ .

Lemma 3.1 (Correlation graph).

Under the incoherence assumption $(A 2)$ and the threshold $ρ$ in assumption $(A 7)$ , the following is true for the edges in the correlation graph $G_{corr (ρ)}$ :

	$\| N_{B} (y_{k}) \cap N_{B} (y_{l}) \| = 1$	$\Rightarrow (y_{k}, y_{l}) \in G_{corr (ρ)}, \forall i \in [r],$		(11)
	$(y_{k}, y_{l}) \in G_{corr (ρ)}$	$\Rightarrow \| N_{B} (y_{k}) \cap N_{B} (y_{l}) \| \geq 1,$		(12)

for all $k, l \in {1, 2,, \dots, n}, k \neq l$ .

Lemma 3.1 suggests that nodes which intersect in exactly one dictionary element are special, in that they are guaranteed to have an edge between them in $G_{corr (ρ)}$ . Our next lemma works towards establishing something even stronger. We will next establish that there are large cliques in the correlation graph where any two samples in the clique intersect in the same unique dictionary element. In order to state the lemma, we need some additional notation.

For each dictionary element $a_{i}$ , consider a set of samples⁴⁴4Note that such a set need not be unique. ${y_{k}, k \in S}$ , for some $S \subset {1, 2, \dots, n}$ , such that they only have $a_{i}$ in common, and denote such a set by $C_{i}$ i.e.

C_{i} := {y_{k}, k \in S : N_{B} (y_{k}) \cap N_{B} (y_{l}) = {a_{i},} \forall k, l \in S} .

(13)

Lemma 3.1 implies that in the correlation graph, the set of nodes in $C_{i}$ form a clique (not necessarily maximal), for each $i \in {1, 2, \dots, r}$ , as shown in Figure (b)b. The above implication can be exploited for recovery of dictionary elements: if we find the set $C_{i}$ , then we can hope to recover the element $a_{i}$ , since that is the only element in common to the samples in $C_{i}$ .

For ease of stating the next lemma, we further define two shorthand notations.

U n i q - i n t e r s e c t (y_{i}, y_{j}) := {(y_{i}, y_{j}) \in G_{corr (ρ)} and | N_{B} (y_{i}) \cap N_{B} (y_{j}) | = 1},

(14)

Intuitively, the samples satisfying $U n i q - i n t e r s e c t (y_{i}, y_{j})$ are guaranteed to have an edge between them by Lemma 3.1. In order to guarantee large cliques, we will also need to measure the number of triangles in $G_{corr (ρ)}$ .

In order to do this, given anchor samples $y_{i^{*}}$ and $y_{j^{*}}$ have a unique intersection, we now bound the probability that a randomly chosen sample $y_{i}$ , among the neighborhood set of $y_{i^{*}}$ and $y_{j^{*}}$ in the correlation graph also has a unique intersection. Now define unique intersection event for a new sample $y_{i}$ with respect to anchor samples $y_{i^{*}}$ and $y_{j^{*}}$ as follows

U n i q - i n t e r s e c t (y_{i}; y_{i^{*}}, y_{j^{*}}) := {N_{B} (y_{i}) \cap N_{B} (y_{i^{*}}) = N_{B} (y_{i}) \cap N_{B} (y_{j^{*}}) = {a_{k}}},

(15)

where $a_{k} = N_{B} (y_{i^{*}}) \cap N_{B} (y_{j^{*}})}$ is the unique intersection of the anchor samples $y_{i^{*}}$ and $y_{j^{*}}$ . In other words, $U n i q - i n t e r s e c t (y_{i}; y_{i^{*}}, y_{j^{*}})$ indicates the event that the pairwise intersections of the new sample $y_{i}$ with each of the anchors $y_{i^{*}}$ and $y_{j^{*}}$ is unique and equal to the unique intersection of $y_{i^{*}}$ and $y_{j^{*}}$ .

Lemma 3.2 (Formation of clique under good anchor samples).

	$P [U n i q - i n t e r s e c t (y_{i}; y_{i^{}}, y_{j^{}}) ∣ ∣ U n i q - i n t e r s e c t (y_{i^{}}, y_{j^{}}), % and (y_{i}, y_{i^{}}), (y_{i}, y_{j^{}}) \in G_{corr (ρ)}]$
	$\geq 1 - \frac{s^{3}}{r} .$

Lemma 3.2 is crucial for our algorithm. It guarantees that given a pair of good anchor elements—one satisfying unique intersection property—a large fraction of their neighrbors also contain this common dictionary element. Some further arguments can then be made to establish that a large fraction of the neighbors of $y_{i^{*}}$ and $y_{j^{*}}$ also have edges amongst themselves and hence form cliques as defined in Equation 13.

3.2 Correctness of Procedure 1

A key component in our analysis is the correctness of Procedure 1. As we saw in the previous lemmas, it is crucial for a chosen pair of anchor elements to have a unique intersection in order to use them for identifying large cliques $C_{i}$ in $G_{corr (ρ)}$ . Procedure 1 plays a crucial role by providing a verifiable test for whether a pair of anchor elements have a unique intersection or not. Our next two lemmas help us establish that this test is sound with high probability. We first show that two neighbors of a bad anchor pair do not have an edge amongst them with high probability.

Denote the event

Δ (y_{i}, y_{j}, y_{k}) := {(y_{i}, y_{j}), (y_{j}, y_{k}), (y_{i}, y_{k}) \in G_{corr (ρ)}},

i.e., the samples $y_{i}, y_{j}, y_{k}$ form a triangle in the correlation graph.

Lemma 3.3 (Detection of bad anchor samples).

For randomly chosen samples $y_{i}, y_{j}$

P [(y_{i}, y_{j}) \notin G_{corr (ρ)} ∣ Δ (y_{i}, y_{i^{*}}, y_{j^{*}}), Δ (y_{j}, y_{i^{*}}, y_{j^{*}}), \neg U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})] > \frac{1}{16} .

Intuitively, this means that the number of sets $S_{i}$ which will be edges in $G_{corr (ρ)}$ is rather small for an anchor pair with multiple dictionary elements in common. In order for correctness of the procedure, we will in fact need this number to be substantially smaller than that for a good anchor pair. This is indeed the case as we next establish.

Lemma 3.4 (Detection of good anchor samples).

For randomly chosen samples $y_{i}, y_{j}$

P [(y_{i}, y_{j}) \notin G_{corr (ρ)} ∣ Δ (y_{i}, y_{i^{*}}, y_{j^{*}}), Δ (y_{j}, y_{i^{*}}, y_{j^{*}}), U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})] \leq \frac{24 s^{3}}{r} .

Combining the above two lemmas, the correctness of Procedure 1 naturally follows.

Proposition 3.1 (Correctness of Procedure 1).

Suppose $(y_{i^{*}}, y_{j^{*}}) \in G_{corr (ρ)}$ . Suppose that $s^{3} \leq r / 1536$ and $γ \leq 1 / 64$ . Then Algorithm 1 returns the value of $U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})$ correctly with probability greater than $1 - 2 exp (- γ^{2} ¯ ¯ ¯ n)$ .

3.3 Proof of Theorem 2.1

In this section we will put all the pieces together and establish Theorem 2.1. We start by establishing that given a pair of good anchor elements, the SVD step in Algorithm 1 approximately recovers the unique dictionary element in the intersection of the two anchors.

Proposition 3.2 (Accuracy of SVD).

Consider anchor samples $y_{i^{*}}$ and $y_{j^{*}}$ such that $U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})$ is satisfied, and wlog, let $N_{B} (y_{i^{*}}) \cap N_{B} (y_{j^{*}}) = {a_{1}}$ . Recall the definition of $ˆ S$ (25), and further define $ˆ L := \sum_{i \in ˆ S} y_{i} y_{i}^{⊤}$ and $ˆ n = | ˆ S |$ . If $ˆ a$ is the top singular vector of $ˆ L$ , then there exists a universal constant $c$ such that we have:

min z \in {- 1, 1} {∥ ˆ a - z a_{1} ∥}_{2}^{2} < \frac{32 s M^{2}}{m^{2}} (\frac{μ_{1}}{\sqrt{d s}} + \frac{μ_{1}^{2}}{d} + \frac{s^{3}}{r} + α^{2} + \frac{α}{\sqrt{s}}),

with probability greater than $1 - d exp (- c α^{2} ˆ n)$ for $α < 1 / 20$ .

Given the above proposition, the proof of Theorem 2.1 is relatively straightforward. Indeed, the key missing piece is the dependence on the random quantity $| ˆ S |$ in the error probability in Proposition 3.2. We now present the proof.

Proof of Theorem 2.1:

Consider a particular iteration of Algorithm 1. Procedure 1 returns $U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})$ with probability greater than $1 - 2 exp (- γ^{2} | ˆ S | / 2)$ . If $\neg U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})$ , then Algorithm 1 proceeds to the next iteration. Consider the case of $U n i q - i n t e r s e c t (y_{i^{*}}, y_{j^{*}})$ and suppose $N_{B} (y_{i^{*}}) \cap N_{B} (y_{j^{*}}) = {a_{l}}$ . Using Proposition 3.2, with probability greater than $1 - d exp (- c α^{2} | ˆ S |)$ , we have:

{∥ a_{l} - ˆ a ∥}_{2}^{2} < \frac{32 s M^{2}}{m^{2}} (\frac{μ_{1}}{\sqrt{d s}} + \frac{μ_{1}^{2}}{d} + \frac{s^{3}}{r} + α^{2} + \frac{α}{\sqrt{s}}) .

Using Lemma A.4 and Lemma 3.1, we see that $| ˆ S | \geq \frac{n s}{4 r}$ with probability greater than $1 - exp (\frac{- n s}{16 r})$ . Using a union bound over all the iterations (which are at most $n^{2}$ ), the above claims hold for all iterations with probability greater than $1 - n^{2} d exp (\frac{- c α^{2} n s}{r}) - 2 n^{2} exp (\frac{- γ^{2} n s}{8 r}) - n^{2} exp (\frac{- n s}{16 r})$ .

Using Lemma A.4 and Lemma 3.1, with probability greater than $1 - r exp (\frac{- n s}{64 r})$ , for every $l \in [r]$ , there are at least $\frac{n s}{8 r}$ pairs $(i^{*}, j^{*})$ such that $N_{B} (y_{i^{*}}) \cap N_{B} (y_{j^{*}}) = {a_{l}}$ and $(i^{*}, j^{*}) \in G_{corr (ρ)}$ . Lines 9-11 of the algorithm then ensure that there is a unique copy of the approximation to $a_{l}$ dictionary element. Using a union bound now gives the result.

$□$

3.4 Analysis of post-processing step

In this section, we will show how to clean up the approximate recovery of the previous section and obtain exact recovery of the dictionary under Assumption B1. We start by setting up the problem as that of sparse estimation with deterministic noise and describing some guarantees in a general setup. We then specialize these to the assumptions of our problem and present the proof of Theorem 2.2.

3.4.1 Lasso with determinstic noise

Recalling the model (1), we see that each observation $y_{i}$ is generated according to the linear model

y_{i} = A x_{i}, for i = 1, 2, \dots, n,

where $x_{i}$ is a $s$ -sparse vector in $r$ dimensions. If we knew the dictionary $A$ , then this is the usual sparse linear system. Given the knowledge of an approximate dictionary $¯ A$ however, we can rewrite the system as

y_{i} = ¯ A x_{i} + (A - ¯ A) x_{i}      w_{i},

(16)

where $W \in R^{d \times n}$ is the error matrix. Note that the errors in $W$ are not zero mean, or even independent of $¯ A$ unlike typical statistical settings. Under our initialization, however, they are bounded, which we establish subsequently. For the remainder of this section, we assume the following facts about $¯ A$ . Note that this is not an assumption about the model, but a condition on the output of Algorithm 1, which will be proved in the next section.

Assumption C1 (Approximate initialization).

Assume that $¯ A$ is an approximately correct initialization for $A$ , meaning the following hold:

RIP: The $2 s$ -RIP constant of the matrix $¯ A$ , $δ_{2 s} < \frac{1}{7}$ . That is, for every $S \subseteq {1, 2, \dots, r}$ with $| S | \leq 2 s$ , the smallest and largest singular values, $σ_{min}$ and $σ_{max}$ respectively of the $d \times | S |$ matrix ${¯ A}_{S}$ satisfy:

\frac{6}{7} < σ_{min} < σ_{max} < \frac{8}{7} .

Bounded error: $∥ {¯ a}_{i} - a_{i} ∥_{2} \leq ϵ_{A}$ for all $i = 1, 2, \dots, r$ .

Under these general assumptions, we can provide a guarantee on the error incurred in (10) in step (2) of Algorithm 2. While this result has been obtained in many contexts by various authors, we use the following precise form from Candes [13].

Theorem 3.1 (Theorem 1.2 from Candes [13]).

Suppose $y_{i}$ is generated according to the linear model (16), where $x_{i}$ is $s$ -sparse and assume that $δ_{2 s} \leq \sqrt{2} - 1$ . Then the solution to Equation (10) obeys the following, for a universal constant $C_{1}$ ,

∥ {ˆ x}_{i} - x_{i} ∥_{2} \leq C_{1} ∥ w_{i} ∥_{2} .

In particular, $C_{1} = 8.5$ suffices for $δ_{2 s} \leq 0.2$ .

3.4.2 Proof of Theorem 2.2

In order to prove Theorem 2.2, we first establish that under our assumptions, the coefficients $x_{i}$ are exactly recovered in Equation (10). Once this is established, Theorem 2.2 follows in a straightforward manner. We start with a useful proposition.

Proposition 3.3.

Under conditions of Theorem 2.1, assume further that $ϵ_{A} \leq 1 / (20 s)$ for the dictionary returned in Algorithm 1. Then Algorithm 2 guarantees that $_{i} = x_{i}$ for all $i = 1, 2, \dots, n$ .

Proof: We would like to use Theorem 3.1 to show that we recover the coefficients $x_{i}$ correctly in the lasso step (10) of Algorithm 2. In order to do this, we first need to verify Assumption C1 for the dictionary returned by Algorithm 1, and then obtain bounds on the quantity $∥ w_{1} ∥_{2}$ . We start with the former.

Consider any $2 s$ -sparse subset $S$ of $[r]$ . We have:

	$σ_{min} ({¯ A}_{S})$
	$σ_{max} ({¯ A}_{S})$

where $ζ_{1}$ and $ζ_{2}$ follow from Lemma A.5 in Appendix A.6. Since $A_{S}$ is a $d \times 2 s$ matrix, it satisfies that ${∥ ∥ A_{S} - {¯ A}_{S} ∥ ∥}_{F} \leq \sqrt{2 s ϵ_{A}}$ . Given the assumption $ϵ_{A} \leq 1 / (20 s)$ , it immediately follows that the minimum and maximum singular values of ${¯ A}_{s}$ are at least $6 / 7$ and $8 / 7$ respectively, so that we obtain $δ_{2 s} = 1 / 7 < 0.2$ .

This shows that $¯ A$ satisfies Assumption C1. Next we bound the $ℓ_{2}$ norm of the noise vector $w_{i}$ . Again bounding the frobenius norm of the error in the dictionary in the same way as above, we obtain

∥ w_{i} ∥_{2} \leq ∥ (A - ¯ A)_{S_{i}} ∥_{2} ∥ x_{i} ∥_{2} \leq {∥ ∥ (A - ¯ A)_{S_{i}} ∥ ∥}_{F} \sqrt{s} \leq s ϵ_{A},

where $S_{i}$ is the support of $x_{i}$ . Consequently, we obtain from Theorem 3.1 that the output ${ˆ x}_{i}$ of Equation 10 satisfies

∥ {ˆ x}_{i} - x_{i} ∥_{2} \leq C_{1} s ϵ_{A} \leq 9 s ϵ_{A} \leq 9 / 20.

(17)

We now observe that an $ℓ_{2}$ error guarantee is also an $ℓ_{\infty}$ error guarantee. Recall that by the model assumption, each non-zero coefficient of $X$ has an absolute value of $1$ . Since Equation (17) guarantees that the $ℓ_{2}$ error guarantee is no larger than $1 / 2$ , all the coefficients will be uniquely recovered and hence ${ˆ x}_{i} = x_{i}$ .

$□$

Proof of Theorem 2.2:

We are now ready to provide our proof of exact recovery. Based on Proposition 3.3, we only need to verify two things. First is that the initialization $¯ A$ satisfies $ϵ_{A} < 1 / (20 s)$ and the second is that the linear system $Y = A X$ is well-posed when we solve for $A$ . In order to verify the former, we observe that our additional conditions in Assumption B1 guarantee that

	$32 s (\frac{μ_{1}}{\sqrt{d s}} + \frac{μ_{1}^{2}}{d})$	$\leq \frac{1}{1200 s^{2}},$
	$\frac{32 s^{4}}{r}$	$\leq \frac{1}{1200 s^{2}}, and$
	$32 s (α^{2} + \frac{α}{\sqrt{s}})$	$\leq \frac{1}{1200 s^{2}} .$

Hence we obtain from Theorem 2.1 that with probability at least $1 - n^{2} d exp (\frac{- c α^{2} n s}{r}) - 4 max (n^{2}, r) exp (\frac{- n s}{32768 r})$ , $ϵ_{A} < 1 / (20 s)$ . Hence, it only remains to verify that the linear system is well-posed.

According to Lemma A.7 in Appendix A.6, the matrix $E [X X^{T}] = \frac{s}{r} I_{r \times r}$ so that all of its singular values are equal to $s / r$ . We now appeal to Theorem A.1 with $W = X$ , $d = r$ and $u = \sqrt{s}$ . Then we obtain for any $t > 0$ with probability at least $1 - r exp (- c t^{2})$

σ_{min} (X X^{T}) \geq \frac{n s}{r} - n max {\sqrt{\frac{s}{r}} δ, δ^{2}},

where $δ = t \sqrt{s / n}$ . Substituting the value of $δ$ , we obtain the lower bound

	$σ_{min} (X X^{T})$	$\geq \frac{n s}{r} - n max {\sqrt{\frac{s}{r}} t \sqrt{\frac{s}{n}}, t^{2} \frac{s}{n}}$
		$\geq \frac{n s}{r} (1 - t \sqrt{\frac{r}{n}} - \frac{t^{2} r}{n})$
		$= \frac{n s}{4 r},$

for $t = \sqrt{n / (4 r)}$ . This means that the linear system is well-posed with probability at least $1 - r exp (- c n / (4 r))$ . Choosing $c_{0}$ to now be $min (c, 1 / 32768)$ finishes the proof. $□$

4 Discussion and Conclusion

In this paper, we proposed simple and tractable methods for dictionary learning. We present a novel clustering-based approach which can approximately recover the uknown overcomplete dictionary from samples. We also analyzed a simple denoising strategy based on sparse recovery algorithms for reconstructing the dictionary exactly under some simplifying assumptions on the model. In particular, the second step is not tied to the first step in any critical way, and more sophisticated post-processing procedures have since been developed. There is of course, also room for developing better approximate recovery schemes, building on our work.

In the analysis of the clustering step, we provide guarantees when the coefficient matrix is sparse and randomly drawn. In principle, our analysis can be extended to general sparse coefficient matrices and can be cast as a higher-order expansion condition on the coefficient bipartite graph. Similar (and yet not the same) expansion conditions have appeared in other contexts involving learning of overcomplete models. For instance, in [5], Anandkumar et. al. establish that under an expansion condition on the topic-word matrix, unsupervised learning of the model is possible. Here, the hidden topics correspond to dictionary elements, and the observed words correspond to the samples in the dictionary setting.

Finally, our work suggests some natural and interesting directions for future research. While both the steps of our algorithm seem inherently robust to noise, it remains important to quantify the recovery properties when the observations are noisy in future work. Another natural question is raised by the fact that we use only one step of lasso and least squares for exact recovery. Indeed, the subsequent work [1] analyzes a generalization where we perform multiple iterations of lasso followed by subsequent dictionary estimation, and is able to exactly recover the dictionary under a much broader set of conditions. Since our study was motivated by natural applications of dictionary learning in signal processing and machine learning, it would also be interesting to investigate how our provably correct procedures perform compared to the popular heuristic methods.

Acknowledgements

A. Agarwal thanks Yonina Eldar for suggesting the problem to him. A. Anandkumar is supported in part by Microsoft Faculty Fellowship, NSF Career award CCF-1254106, NSF Award CCF-1219234, and ARO YIP Award W911NF-13-1-0084. P. Netrapalli thanks Yash Deshpande for helpful discussions. The authors thank Matus Telgarsky for suggesting Lemma A.6 and thank Sham Kakade and Dean Foster for initial discussions.

Appendix A Proofs for clustering analysis

In this section we will provide the proofs of many of the Lemmas along with some auxilliary results in Sections 3.1- 3.3. Some of the more technical results that are required will be deferred to Appendix A.6.

a.1 Proofs of correlation graph properties

We start by proving Lemmas 3.1 and 3.2 in Section 3.1.

Proof of Lemma 3.1:

We first prove (12) via contradiction. Suppose $N_{B} (y_{k}) \cap N_{B} (y_{l}) = \emptyset$ , we then have

	$\| ⟨ y_{k}, y_{l} ⟩ \|$	$= \| \sum i, j x_{i k} x_{j l} ⟨ a_{i}, a_{j} ⟩ \| \leq \sum i, j \| x_{i k} x_{j l} ⟨ a_{i}, a_{j} ⟩ \|$
		$\leq \| N_{B} (y_{k}) \| \cdot \| N_{B} (y_{l}) \| \cdot max i, j, k, l \| x_{i k} x_{j l} \| \cdot max i \neq j \| ⟨ a_{i}, a_{j} ⟩ \| \leq \frac{s^{2} M^{2} μ_{0}}{\sqrt{d}}$

For (11), let ${a_{i^{*}}} = N_{B} (y_{k}) \cap N_{B} (y_{l})$

| ⟨ y_{k}, y_{l} ⟩ |

= | \sum i, j x_{i k} x_{j l} ⟨ a_{i}, a_{j} ⟩ | \geq | x

A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries

Authors

Learning Stable Multilevel Dictionaries for Sparse Representations

Group K-Means

Fast approximations to structured sparse coding and applications to object classification

Learning Deep Analysis Dictionaries – Part I: Unstructured Dictionaries

Multiscale Residual Mixture of PCA: Dynamic Dictionaries for Optimal Basis Learning

PADDLE: Proximal Algorithm for Dual Dictionaries LEarning

On Learning Sparsely Used Dictionaries from Incomplete Samples

This week in AI

1 Introduction

1.1 Summary of Results

1.2 Related Works

Dictionary Learning:

Blind Source Separation/ICA/Topic Models:

Connection to Learning Overlapping Communities:

1.3 Overview of Techniques

Dictionary estimation via clustering:

Key intuitions for the clustering procedure:

Sparse regression for post-processing:

2 Method and Guarantees

Notation:

2.1 Clustering procedure and its analysis

2.1.1 Algorithm

2.1.2 Assumptions and Main Result

Assumptions:

Main Result:

Theorem 2.1 (Approximate recovery of dictionary).

Remark:

2.2 Post-processing for binary coefficients

2.2.1 Algorithm

2.2.2 Exact recovery for bernoulli coefficients

Assumption B1 (Conditions for exact recovery).

Theorem 2.2 (Exact recovery for bernoulli coefficients).

Remark:

3 Proofs of main results

3.1 Correlation graph properties

Lemma 3.1 (Correlation graph).

Lemma 3.2 (Formation of clique under good anchor samples).

3.2 Correctness of Procedure 1

Lemma 3.3 (Detection of bad anchor samples).

Lemma 3.4 (Detection of good anchor samples).

Proposition 3.1 (Correctness of Procedure 1).

3.3 Proof of Theorem 2.1

Proposition 3.2 (Accuracy of SVD).

3.4 Analysis of post-processing step

3.4.1 Lasso with determinstic noise

Assumption C1 (Approximate initialization).

Theorem 3.1 (Theorem 1.2 from Candes [13]).

3.4.2 Proof of Theorem 2.2

Proposition 3.3.

4 Discussion and Conclusion

Acknowledgements

Appendix A Proofs for clustering analysis

a.1 Proofs of correlation graph properties