When is sparse dictionary learning well-posed?

I Introduction

Blind source separation is a classical problem in signal processing [1]. In one common modern formulation, each of $N$ observed $n$ -dimensional signals is a (noisy) linear combination of at most $k$ elementary waveforms drawn from some unknown dictionary of size $m$ , typically with $m ≪ N$ (see [2] for a comprehensive review of this and related models). Approximating solutions to this sparsity-constrained inverse problem have provided insight into the structure of many signal classes lacking domain-specific formal models (e.g., in vision [3]

). In particular, it has been shown that response properties of simple-cell neurons in mammalian visual cortex emerge from optimizing a dictionary to represent small patches of natural images

[4, 5, 6, 7], a major advance in computational neuroscience. A curious aspect of this finding is that the latent waveforms (e.g., “Gabor” wavelets) estimated from data appear to be canonical [8]; i.e., they are found in learned dictionaries independent of algorithm or image training set.

Motivated by these discoveries and earlier work in the theory of neural communication [9], we address when dictionaries and the sparse representations they induce are uniquely determined by data. Answers to this question also have other real-world implications. For example, a sparse coding analysis of local painting style can be used for forgery detection [10], but only if all dictionaries consistent with training data do not differ appreciably in their ability to sparsely encode new samples. Fortunately, algorithms with proven recovery of generating dictionaries under certain conditions have recently been proposed (see [11, Sec. I-E] for a summary of the state-of-the-art). Few theorems, however, can be cited to explain this uniqueness more broadly; in particular, a universal guarantee in the context of noise has yet to emerge.

Here, we prove very generally that uniqueness and stability is an expected property of the sparse linear coding model. More specifically, dictionaries that preserve sparse codes (e.g., satisfy a “spark condition”) are identifiable from as few as $N = m (k - 1) (\frac{m}{k}) + m$ noisy sparse linear combinations of their columns up to an error that is linear in the noise (Thm. 1). In fact, provided $n \geq min (2 k, m)$ , in almost all cases the dictionary learning problem is well-posed (as per Hadamard [12]) given enough data (Cor. 2). Moreover, these explicit, algorithm-independent guarantees hold without assuming the recovered matrix satisfies a spark condition, and even when the number $m$ of dictionary elements is unknown.

More formally, let $A \in R^{n \times m}$ be a matrix with columns $A_{j}$ ( $j = 1, \dots, m$ ) and let dataset $Z$ consist of measurements:

z_{i} = A x_{i} + n_{i}, i = 1, \dots, N,

(1)

for $k$ -sparse $x_{i} \in R^{m}$ having at most $k < m$ nonzero entries and noise $n_{i} \in R^{n}$ , with bounded norm $∥ n_{i} ∥_{2} \leq η$ representing our combined worst-case uncertainty in measuring $A x_{i}$ . The first mathematical problem we address is the following.

Problem 1 (Sparse linear coding).

Find a matrix $B$ and $k$ -sparse ${¯ ¯ ¯ x}_{1}, \dots, {¯ ¯ ¯ x}_{N} \in R^{¯ ¯¯ ¯ m}$ so that $∥ z_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2} \leq η$ for all $i$ .

Note that any particular solution $(B, {¯ ¯ ¯ x}_{1} \dots, {¯ ¯ ¯ x}_{N})$ to this problem gives rise to an orbit of equivalent solutions $(B P D, D^{- 1} P^{⊤} {¯ ¯ ¯ x}_{1}, \dots, D^{- 1} P^{⊤} {¯ ¯ ¯ x}_{N})$ , where $P$ is any permutation matrix and $D$ any invertible diagonal. Previous theoretical work addressing the noiseless case $η = 0$ (e.g., [13, 14, 15, 16]) with $¯ ¯¯¯ ¯ m = m$ has shown that a solution to Prob. 1 (when it exists) is unique up to this ambiguity provided the $x_{i}$ are sufficiently diverse and the matrix $A$ satisfies the spark condition:

A x_{1} = A x_{2} ⟹ x_{1} = x_{2}, for all k -sparse x_{1}, x_{2},

(2)

which would be, in any case, necessary for uniqueness of the $x_{i}$ . Here, we study stability in the practical setting of noise.

Definition 1.

Fix $Y = {y_{1}, \dots, y_{N}} \subset R^{n}$ . We say $Y$ has a $k$ -sparse representation in $R^{m}$ if there exists a matrix $A$ and $k$ -sparse $x_{1}, \dots, x_{N} \in R^{m}$ such that $y_{i} = A x_{i}$ for all $i$ . This representation is stable if for every $δ_{1}, δ_{2} \geq 0$ , there exists some $ε = ε (δ_{1}, δ_{2})$ that is strictly positive for positive $δ_{1}$ and $δ_{2}$ such that if $B$ and $k$ -sparse ${¯ ¯ ¯ x}_{1}, \dots, {¯ ¯ ¯ x}_{N} \in R^{m}$ satisfy:

∥ A x_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2} \leq ε (δ_{1}, δ_{2}), for all i = 1, \dots, N,

then there is some permutation matrix $P$ and invertible diagonal matrix $D$ such that for all $i, j$ :

∥ A_{j} - {B P D}_{j} ∥_{2} \leq δ_{1} % and ∥ x_{i} - D^{- 1} P^{⊤} {¯ ¯ ¯ x}_{i} ∥_{1} \leq δ_{2} .

(3)

To see how Def. 1 directly relates to Prob. 1, suppose that $Y$ has a stable $k$ -sparse representation in $R^{m}$ and fix $δ_{1}, δ_{2}$ to be the desired recovery accuracy in (3). Consider any dataset $Z$ generated as in (1) with $η \leq \frac{1}{2} ε (δ_{1}, δ_{2})$ . Then from the triangle inequality, it follows that any matrix $B$ and $k$ -sparse ${¯ ¯ ¯ x}_{1}, \dots, {¯ ¯ ¯ x}_{N} \in R^{m}$ solving Prob. 1 are necessarily within $δ_{1}, δ_{2}$ of the original dictionary $A$ and codes $x_{i}$ , respectively.

In the next section, we give precise statements of our main results, which include an explicit form for $ε (δ_{1}, δ_{2})$ . We then prove our main theorem (Thm. 1) in Sec. III after stating some additional definitions and lemmas required for the proof, including a useful result in combinatorial matrix analysis (Lem. 1, proven in Appendix). We also provide an argument that extends our guarantees to the following more common optimization formulation of the dictionary learning problem (Thm. 2).

Problem 2.

Find $B$ and ${¯ ¯ ¯ x}_{1}, \dots, {¯ ¯ ¯ x}_{N} \in R^{¯ ¯¯ ¯ m}$ that solve:

min N \sum i = 1 ∥ {¯ ¯ ¯ x}_{i} ∥_{0} % subject to ∥ z_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2} \leq η, for all i .

(4)

We then sketch proofs of probabilistic extensions of Thm. 1 to random data and dictionaries (Thm. 3 and Cor. 2). Finally, in Sec. IV, we discuss both theoretical and practical applications of our main mathematical findings.

Ii Results

Before stating our results precisely, we identify criteria on the support sets of the generating codes $x_{i}$ that imply stable sparse representations. Letting ${1, \dots, m}$ be denoted $[m]$ , its power set $2^{[m]}$ , and $(\frac{[m]}{k})$ the set of subsets of $[m]$ of size $k$ , we say a hypergraph $H \subseteq 2^{[m]}$ on vertices $[m]$ is $k$ -uniform when $H \subseteq (\frac{[m]}{k})$ . The degree ${deg}_{H} (i)$ of a node $i \in [m]$ is the number of sets in $H$ that contain $i$ , and we say $H$ is regular when for some $r$ we have ${deg}_{H} (i) = r$ for all $i$ (given such an $r$ , we say $H$ is $r$ -regular). We also write $2 H := {S \cup S^{'} : S, S^{'} \in H}$ .

Definition 2.

Given $H \subseteq 2^{[m]}$ , the star $σ (i)$ is the collection of sets in $H$ containing $i$ . We say $H$ has the singleton intersection property (SIP) when $\cap σ (i) = {i}$ for all $i \in [m]$ .

Next, we describe a quantitative spark condition. The lower bound of a matrix $M \in R^{n \times m}$ is the largest $α$ with $∥ M x ∥_{2} \geq α ∥ x ∥_{2}$ for all $x \in R^{m}$ [17]. By compactness of the unit sphere, injective linear maps have nonzero lower bound; hence, if $M$ satisfies (2), then each submatrix formed from $2 k$ of its columns or less has a strictly positive lower bound.

We generalize the lower bound to a domain-restricted union of subspaces model [18] derived from a hypergraph $H \subseteq (\frac{[m]}{k})$ . Let $M_{S}$ denote the submatrix formed by the columns of $M$ indexed by $S \subseteq [m]$ , with $M_{\emptyset} := 0$ . (In the sections that follow, we write $M_{S}$ to denote the column-span of a submatrix $M_{S}$ , and $M_{G}$ to denote ${M_{S}}_{S \in G}$ .) Define:

L_{H} (M) := \frac{1}{\sqrt{k}} min {\frac{∥ M_{S} x ∥_{2}}{∥ x ∥_{2}} : S \in H, x \in R^{| S |}},

(5)

where we write $L_{k}$ when $H = (\frac{[m]}{k})$ .¹¹1We note that $1 - \sqrt{k} L_{k} (M)$ is known as the asymmetric lower restricted isometry constant for matrices $M$ with unit $ℓ_{2}$ -norm columns [19]. Clearly, $L_{2 k} (M) > 0$ for $M$ satisfying (2) and $L_{k^{'}} (M) \geq L_{k} (M)$ whenever $k^{'} \leq k$ . Note also that $L_{2} \geq L_{2 H} \geq L_{2 k}$ if $H \subseteq (\frac{[m]}{k})$ has $\cup H = [m]$ .

A vector

x

is said to be supported in

S \subseteq [m]

when

x∈\rm span{ej:j∈S}

, where

e_{j}

form the standard column basis in

R^{m}

. A set of

k

-sparse vectors is said to be in general linear position when any

k

of them are linearly independent. The following is a precise statement of our main result. We leave the quantity

C_{1} = C_{1} (A, {x_{i}}_{i = 1}^{N}, H)

undefined until Eq. (15). All of our theorems assume matrices consist of real numbers.

Theorem 1.

Fix an $n \times m$ matrix $A$ with $L_{2 H} (A) > 0$ for an $r$ -regular $H \subseteq (\frac{[m]}{k})$ with the SIP. If ${x_{1}, \dots, x_{N}} \subset R^{m}$ contains, for each $S \in H$ , more than $(k - 1) (\frac{m}{k})$ $k$ -sparse vectors in general linear position supported in $S$ , then there is $C_{1} > 0$ with the following holding for all²²2The condition $ε < L_{2} (A) / C_{1}$ is necessary; otherwise, with $A$ = $I$ and $x_{i} = e_{i}$ , the matrix and 1-sparse codes ${¯ ¯ ¯ x}_{i} = e_{2}$ for $i = 1, 2$ and ${¯ ¯ ¯ x}_{i} = e_{i}$ for $i \geq 3$ satisfy $∥ A x_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2} \leq ε$ but nonetheless violate (6). $ε < L_{2} (A) / C_{1}$ :

Every $n \times ¯ ¯¯¯ ¯ m$ matrix $B$ for which there are $k$ -sparse ${¯ ¯ ¯ x}_{1}, \dots, {¯ ¯ ¯ x}_{N}$ satisfying $∥ A x_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2} \leq ε$ for all $i$ has $¯ ¯¯¯ ¯ m \geq m$ and, provided $(r - 1) ¯ ¯¯¯ ¯ m < m r$ ,

∥ A_{j} - B_{¯ ¯ ¯ J} {P D}_{j} ∥_{2} \leq C_{1} ε, for all j \in J,

(6)

for some nonempty $¯ ¯¯ ¯ J \subseteq [¯ ¯¯¯ ¯ m]$ and $J \subseteq [m]$ of size $¯ ¯¯¯ ¯ m - r (¯ ¯¯¯ ¯ m - m)$ , permutation matrix $P$ , and invertible diagonal matrix $D$ .

Moreover, if $A$ satisfies (2) and $ε < L_{2 k} (A) / C_{1}$ , then $L_{2 k} (B P D) \geq L_{2 k} (A) - C_{1} ε$ and:

∥ x_{i} - {(D^{- 1} P^{⊤})}_{¯ ¯ ¯ J}^{- 1} {¯ ¯ ¯ x}_{i} ∥_{1}

\leq (\frac{1 + C_{1} ∥ x_{i} ∥_{1}}{L_{2 k} (A) - C_{1} ε}) ε, for i \in [N],

(7)

where $x_{i}$ and ${¯ ¯ ¯ x}_{i}$ here represent subvectors formed from restricting to entries indexed by $J$ and $¯ ¯¯ ¯ J$ , respectively.

In words, Thm. 1 says that the smaller the difference $¯ ¯¯¯ ¯ m - m$ between recovery and original latent dictionary sizes, the more columns and coefficients of the original dictionary $A$ and codes $x_{i}$ are contained (up to noise) in the appropriately scaled dictionary $B$ and codes ${¯ ¯ ¯ x}_{i}$ . In the particular case when $¯ ¯¯¯ ¯ m = m$ , the theorem directly implies that $Y = {{A x}_{1}, \dots, {A x}_{N}}$ has a stable $k$ -sparse representation in $R^{m}$ , with inequalities (3) guaranteed for $ε$ in Def. 1 given by:

ε (δ_{1}, δ_{2}) := min {\frac{δ_{1}}{C_{1}}, \frac{δ_{2} L_{2 k} (A)}{1 + C_{1} (δ_{2} + {max}_{i \in [N]} ∥ x_{i} ∥_{1})}} .

(8)

Note that sparse codes $x_{i}$ with a shared support that are in general linear position are straightforward to produce using a “Vandermonde” matrix construction (i.e., use the columns of the matrix $[γ_{i}^{j}]_{i, j = 1}^{k, N}$ , for distinct nonzero $γ_{i}$ ). Thus, the assumptions of Thm. 1 are easily met, leading to the following direct application to uniqueness in sparse linear coding.

Corollary 1.

Given a regular hypergraph $H \subseteq (\frac{[m]}{k})$ with the SIP, there are $N = | H | [(k - 1) (\frac{m}{k}) + 1]$ vectors $x_{1}, \dots, x_{N} \in R^{m}$ such that every matrix $A$ with $L_{2 k} (A) > 0$ generates a dataset $Y = {A x_{1}, \dots, A x_{N}}$ with a stable $k$ -sparse representation in $R^{m}$ (with $ε (δ_{1}, δ_{2})$ as in (8)).

One can also easily verify that for every $k < m$ , there are regular $k$ -uniform hypergraphs with the SIP besides the obvious $H = (\frac{[m]}{k})$ ; for instance, take $H$ to be the consecutive intervals of length $k$ in some cyclic order on $[m]$ . In this case, a direct consequence of Cor. 1 is rigorous verification of the lower bound $N = m (k - 1) (\frac{m}{k}) + m$ for sufficient sample size from the introduction. Often, the SIP is achievable with fewer supports. For example, when $k = \sqrt{m}$ , take $H$ to be the $2 k$ rows and columns formed by arranging $[m]$ into a square grid.

Another practical implication of Thm. 1 is the following: there is an effective procedure sufficient to affirm if a proposed solution to Prob. 1 is indeed unique (up to noise and inherent ambiguities). Simply check that the matrix and codes satisfy the (computable) assumptions of Thm. 1 on $A$ and the $x_{i}$ .

We furthermore note that unlike in previous works, the matrix $A$ need not satisfy (2) to be recoverable from data. As an example for $k = 2$ , let $A = [e_{1}, \dots, e_{5}, v]$ where $v = e_{1} + e_{3} + e_{5}$ , and take $H$ to be all consecutive pairs of $[m]$ arranged in cyclic order. Then, dictionary $A$ satisfies the assumptions of Thm. 1 underlying (6) without satisfying (2).

There are other less direct consequences of Thm. 1. For instance, we use it to prove uniqueness for the optimization formulation of sparse linear coding, Prob. 2, the main object of interest for those applying dictionary learning to their data.

Theorem 2.

If the assumptions of Thm. 1 hold, only now with more than $(k - 1) [(\frac{¯ ¯¯ ¯ m}{k}) + | H | k (\frac{¯ ¯¯ ¯ m}{k - 1})]$ vectors $x_{i}$ supported in each $S \in H$ , then all solutions to Prob. 2 with $η \leq ε / 2$ necessarily satisfy recovery inequalities (6) and (7) of Thm. 1.

Another extension of Thm. 1 arises from the following analytic characterization of the spark condition. Let $A$ be the $n \times m$ matrix of $n m$ indeterminates $A_{i j}$ . When real numbers are substituted for $A_{i j}$ , the resulting matrix satisfies (2) if and only if the following polynomial is nonzero:

f (A) := \prod S \in (\frac{[m]}{2 k}) \sum S^{'} \in (\frac{[n]}{2 k}) (det A_{S^{'}, S})^{2},

where for any $S^{'} \in (\frac{[n]}{2 k})$ and $S \in (\frac{[m]}{2 k})$ , the symbol $A_{S^{'}, S}$ denotes the submatrix of entries $A_{i j}$ with $(i, j) \in S^{'} \times S$ . We note that the large number of terms in this product is likely necessary due to the NP-hardness of deciding whether a given matrix $A$ satisfies the spark condition [20].

Since $f$ is analytic, having a single substitution of a real matrix $A$ with $f (A) \neq 0$ necessarily implies that the zeroes of $f$ form a set of (Borel) measure zero. Fortunately, such a matrix $A$ is easily constructed by adding rows of zeroes to any $min (2 k, m) \times m$ Vandermonde matrix as described above (so that each term in the product above for $f$ is nonzero). Hence, almost every $n \times m$ matrix with $n \geq min (2 k, m)$ satisfies (2).

A similar phenomenon applies to datasets of vectors with a stable sparse representation. As in [16, Sec. IV], consider the “symbolic” dataset $Y = {A x_{1}, \dots, A x_{N}}$ generated by indeterminate $A$ and indeterminate $k$ -sparse $x_{1}, \dots, x_{N}$ .

Theorem 3.

There is a polynomial $g$ in the entries of $A \in R^{n \times m}$ and $x_{i} \in R^{m}$ with the following property: if $g$ evaluates to a nonzero number and more than $(k - 1) (\frac{m}{k})$ of the resulting $x_{i}$ are supported in each $S \in H$ for some regular $H \subseteq (\frac{[m]}{k})$ with the SIP, then $Y$ has a stable $k$ -sparse representation in $R^{m}$ (Def. 1). In particular, all – except for a Borel set of measure zero – substitutions impart to $Y$ this property.

Corollary 2.

Fix $m > k$ , $n \geq min (2 k, m)$ , and let the entries of $A \in R^{n \times m}$ and $k$ -sparse $x_{1}, \dots, x_{N} \in R^{m}$

be drawn independently from probability measures absolutely continuous with respect to the standard Borel measure. If more than

(k - 1) (\frac{m}{k})

of the vectors

x_{i}

are supported in each

S \in H

for a regular

H \subseteq (\frac{[m]}{k})

with the SIP, then

Y

has a stable

k

-sparse representation in

R^{m}

with probability one.

Thus, choosing the dictionary and codes “randomly” almost certainly generates data with a stable sparse representation.

We remark that these results have an important application to theoretical neuroscience by mathematically justifying one of the few hypothesized theories of bottleneck communication between sparsely active neural populations [9].

Proposition 1.

Sparse neural population activity is recoverable from noisy random linear compression by any method that solves Prob. 1 or 2; in particular, via biophysically plausible unsupervised sparse linear coding (e.g., [21, 22, 23]).

We close this section with comments on optimality. Our linear scaling for $ε$ in (8) is essentially optimal (e.g., see [24]), but a basic open problem remains: how many samples are necessary to determine the sparse linear coding model? If $k$ is held fixed or if the size of the support set of reconstructing codes is known to be polynomial in $¯ ¯¯¯ ¯ m$ and $k$ , then a practical (polynomial) amount of data suffices.³³3In the latter case, a reexamination of the pigeonholing argument in the proof of Thm. 1 requires a polynomial number of samples distributed over a polynomial number of supports. Reasons to be skeptical that this holds in general, however, can be found in [20, 25].

Iii Proofs

We now begin our proof of Thm. 1 by showing how dictionary recovery (6) already implies sparse code recovery (7) when $ε < L_{2 k} (A) / C_{1}$ (provided $A$ satisfies (2)), temporarily assuming (without loss of generality) that $¯ ¯¯¯ ¯ m = m$ . For $2 k$ -sparse $x \in R^{m}$ , the triangle inequality gives $∥ (A - B P D) x ∥_{2} \leq C_{1} ε ∥ x ∥_{1} \leq C_{1} ε \sqrt{2 k} ∥ x ∥_{2}$ . Thus:

	$∥ B P D x ∥_{2}$	$\geq \| ∥ A x ∥_{2} - ∥ (A - B P D) x ∥_{2} \|$
		$\geq \sqrt{2 k} (L_{2 k} (A) - C_{1} ε) ∥ x ∥_{2},$

since $ε < L_{2 k} (A) / C_{1}$ . Hence, $L_{2 k} (B P D) \geq L_{2 k} (A) - C_{1} ε > 0$ , and (7) then follows from:

	$∥ x_{i} - D^{- 1} P^{⊤} {¯ ¯ ¯ x}_{i} ∥_{1}$	$\leq \frac{∥ B P D (x_{i} - D^{- 1} P^{⊤} {¯ ¯ ¯ x}_{i}) ∥_{2}}{L_{2 k} (B P D)}$
		$\leq \frac{∥ (B P D - A) x_{i} ∥_{2} + ∥ A x_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2}}{L_{2 k} (B P D)}$
		$\leq \frac{ε (1 + C_{1} ∥ x_{i} ∥_{1})}{L_{2 k} (B P D)} .$

The heart of the matter is therefore (6), which we now establish, first in the important special case of $k = 1$ .

Proof of Thm. 1 for $k = 1$ .

Since the only 1-uniform hypergraph with the SIP is $[m]$ , we have $x_{i} = c_{i} e_{i}$ for $c_{i} \in R ∖ {0}$ , $i \in [m]$ . In this case, we require that $C_{1} \geq 1 / {min}_{ℓ \in [m]} | c_{ℓ} |$ .

Fix $A \in R^{n \times m}$ satisfying (2) and suppose that for some $B$ and $1$ -sparse ${¯ ¯ ¯ x}_{i} \in R^{¯ ¯¯ ¯ m}$ we have $∥ A x_{i} - B {¯ ¯ ¯ x}_{i} ∥_{2} \leq ε < L_{2} (A) / C_{1}$ for all $i$ . Then, there exist ${¯ ¯ c}_{1}, \dots, {¯ ¯ c}_{m} \in R$ and a map $π : [m] \to [¯ ¯¯¯ ¯ m]$ such that:

∥ c_{j} A_{j} - {¯ ¯ c}_{j} B_{π (j)} ∥_{2} \leq ε, for j \in [m] .

(9)

Note that ${¯ ¯ c}_{j} \neq 0$ , since otherwise we have the contradiction $∥ A_{j} ∥_{2} \leq C_{1} | c_{j} | ∥ A_{j} ∥_{2} < L_{2} (A) \leq L_{1} (A) = {min}_{j \in [m]} ∥ A_{j} ∥_{2}$ .

We now show that $π$ is injective (in particular, a permutation if $¯ ¯¯¯ ¯ m = m$ ). Suppose that $π (i) = π (j) = ℓ$ for some $i \neq j$ and $ℓ$ . Then, $∥ c_{j} A_{j} - {¯ ¯ c}_{j} B_{ℓ} ∥_{2} \leq ε$ and $∥ c_{i} A_{i} - {¯ ¯ c}_{i} B_{ℓ} ∥_{2} \leq ε$ . Scaling and summing these inequalities by $| {¯ ¯ c}_{i} |$ and $| {¯ ¯ c}_{j} |$ , respectively, and applying the triangle inequality, we obtain:

	$(\| {¯ ¯ c}_{i} \| + \| {¯ ¯ c}_{j} \|) ε$	$\geq ∥ A ({¯ ¯ c}_{i} c_{j} e_{j} - {¯ ¯ c}_{j} c_{i} e_{i}) ∥_{2}$
		$\geq (\| {¯ ¯ c}_{i} \| + \| {¯ ¯ c}_{j} \|) L_{2} (A) min ℓ \in [m] \| c_{ℓ} \|,$

which contradicts the bound $ε < L_{2} (A) / C_{1}$ . Hence, the map $π$ is injective and therefore $¯ ¯¯¯ ¯ m \geq m$ . Setting $¯ ¯¯ ¯ J = π ([m])$ and letting $P = (e_{π (1)} \dots e_{π (m)})$ and $D = diag (\frac{{¯ c}_{1}}{c_{1}}, \dots, \frac{{¯ c}_{m}}{c_{m}})$ , we see that (9) becomes, for all $j \in [m]$ :

∥ A_{j} - B_{¯ ¯ ¯ J} {P D}_{j} ∥_{2} = ∥ A_{j} - \frac{{¯ ¯ c}_{j}}{c_{j}} B_{π (j)} ∥_{2} \leq \frac{ε}{| c_{j} |} \leq C_{1} ε .

∎

We require a few additional tools to extend the proof to the general case $k < m$ . These include a generalized notion of distance (Def. 3) and angle (Def. 4) between subspaces as well as a stability result in combinatorial matrix analysis (Lem. 1).

Definition 3.

For $u \in R^{m}$ and vector spaces $U, V \subseteq R^{m}$ , let $\rm dist(u,V):=min{∥u−v∥2:v∈V}$ and define:

d(U,V):=maxu∈U, ∥u∥2≤1% \rm dist(u,V).

(10)

We note the following facts. If $U^{'} \subseteq U$ , then $d (U^{'}, V) \leq d (U, V)$ and [26, Ch. 4 Cor. 2.6]:

d (U, V) < 1 ⟹ dim (U) \leq dim (V) .

(11)

Also, from [27, Lem. 3.2], we have:

dim (U) = dim (V) ⟹ d (U, V) = d (V, U) .

(12)

Our result in combinatorial matrix analysis is the following.

Lemma 1.

Suppose the $n \times m$ matrix $A$ has $L_{2 H} (A) > 0$ for some $r$ -regular $H \subseteq (\frac{[m]}{k})$ with the SIP. There exists $C_{2} > 0$ for which the following holds for all $ε < L_{2} (A) / C_{2}$ :

If for some $n \times ¯ ¯¯¯ ¯ m$ matrix $B$ and map $π : H \mapsto (\frac{[¯ ¯¯ ¯ m]}{k})$ ,

d (A_{S}, B_{π (S)}) \leq ε, for S \in H;

(13)

then $¯ ¯¯¯ ¯ m \geq m$ , and, provided $¯ ¯¯¯ ¯ m < m r / (r - 1)$ , there is a permutation matrix $P$ and invertible diagonal $D$ such that:

∥ A_{j} - B_{¯ ¯ ¯ J} {P D}_{j} ∥_{2} \leq C_{2} ε, for j \in J,

(14)

for some nonempty $¯ ¯¯ ¯ J \subseteq [¯ ¯¯¯ ¯ m]$ and $J \subseteq [m]$ of size $¯ ¯¯¯ ¯ m - r (¯ ¯¯¯ ¯ m - m)$ .

The constant $C_{1} > 0$ in Thm. 1 is then given by⁴⁴4Note that $∥ {A X}_{I (S)} c ∥_{2} \geq \sqrt{k} L_{H} (A) ∥ X_{I (S)} c ∥_{2} \geq k L_{H} (A) L_{k} (X_{I (S)}) ∥ c ∥_{2}$ for $S \in H$ and $k$ -sparse $c$ . Therefore, $L_{k} ({A X}_{I (S)}) \geq \sqrt{k} L_{H} (A) L_{k} (X_{I (S)}) > 0$ since $L_{2 H} (A) > 0$ and $L_{k} (X_{I (S)}) > 0$ by general linear position of the $x_{i}$ . Thus, $C_{1} > 0$ .:

C_{1} (A, {x_{i}}_{i = 1}^{N}, H) := \frac{C_{2} (A, H)}{{min}_{S \in H} L_{k} ({A X}_{I (S)})},

(15)

where, given vectors $x_{1}, \dots, x_{N} \in R^{m}$ , we denote by $X$ the $m \times N$ matrix with columns $x_{i}$ and by $I (S)$ the set of indices $i$ for which $x_{i}$ is supported in $S$ .

The constant $C_{2} = C_{2} (A, H)$ , in turn, will be presented relative to a quantity from [28] used to analyze the convergence of the alternating projections algorithm. Specifically, in Lem. 3 below, the following definition is used to bound the distance between a point and the intersection of subspaces given an upper bound on its distance from each individual subspace.

Definition 4.

For a collection of real subspaces $V = {V_{i}}_{i = 1}^{ℓ}$ , define $ξ^{2} := 0$ when $| V | = 1$ , and otherwise:

ξ^{2} (V) := 1 - max ℓ - 1 \prod i = 1 {sin}^{2} θ (V_{i}, \cap_{j > i} V_{j}),

(16)

where the maximum is taken over all orderings of the $V_{i}$ and the angle $θ \in (0, \frac{π}{2}]$ is defined implicitly as [28, Def. 9.4]:

cos θ (U, W) := max {| ⟨ u, w ⟩ | : \begin{matrix} u \in U \cap (U \cap W)^{⊥}, ∥ u ∥_{2} \leq 1 w \in W \cap (U \cap W)^{⊥}, ∥ w ∥_{2} \leq 1 \end{matrix}} .

Note that $θ \in (0, \frac{π}{2}]$ implies $0 \leq ξ < 1$ , and that $ξ (V^{'}) \leq ξ (V)$ when $V^{'} \subseteq V$ .⁵⁵5We acknowledge the counter-intuitive property that $θ = π / 2$ when $U \subseteq W$ . The constant $C_{2}$ in Lem. 1 is then:

C_{2} (A, H) := \frac{(r + 1) {max}_{j \in [m]} ∥ A_{j} ∥_{2}}{1 - {max}_{G \in (\frac{H}{r + 1})} ξ (A_{G})},

(17)

which we remark yields a constant $C_{1}$ consistent with what is required for the case $k = 1$ considered at the beginning of this section.⁶⁶6For $x_{i} = c_{i} e_{i}$ , the denominator in (15) becomes ${min}_{i \in [m]} | c_{i} | ∥ A_{i} ∥_{2}$ , hence with $r = 1$ we have $C_{1} \geq 2 / {min}_{i \in [m]} | c_{i} |$ .

The pragmatic reader should note that the explicit constants $C_{1}$ and $C_{2}$ are effectively computable: the quantity $L_{k}$

may be calculated as the smallest singular value of a certain matrix, while the quantity

ξ

involves computing “canonical angles” between subspaces, which reduce again to an efficient singular value decomposition. There is no known fast computation of

L_{k}

in general, however, since even

L_{k} > 0

is NP-hard [20]; although fixing

k

yields polynomial complexity. Moreover, calculating

C_{2}

requires an exponential number of queries to

ξ

unless

r

is held fixed, too (e.g., the “cyclic order” hypergraphs described above have

r = k

). Thus, as presented,

C_{1}

and

C_{2}

are not efficiently computable.

Proof of Thm. 1 for $k < m$ .

We find a map $π : H \to 2^{[¯ ¯¯ ¯ m]}$ for which the distance $d (A_{S}, B_{π (S)})$ is controlled by $ε$ . Applying Lem. 1 then completes the proof.

Since there are more than $(k - 1) (\frac{¯ ¯¯ ¯ m}{k})$ vectors $x_{i}$ supported in each $S \in H$ , the pigeonhole principle gives $¯ ¯¯ ¯ S \in (\frac{[¯ ¯¯ ¯ m]}{k})$ and a set of $k$ indices $K \subseteq I (S)$ with all ${¯ ¯ ¯ x}_{i}$ , $i \in K$ , supported in $π (S) := ¯ ¯¯ ¯ S$ . It also follows from $L_{2 H} (A) > 0$ and the general linear position of the $x_{i}$ that $L_{k} ({A X}_{K}) > 0$ ; that is, the columns of the $n \times k$ matrix ${A X}_{K}$ form a basis for $A_{S}$ .

Fixing $0 \neq y \in A_{S}$ , there then exists $0 \neq c = (c_{1}, \dots, c_{k}) \in R^{k}$ such that $y = {A X}_{K} c$ . Setting $¯ ¯ ¯ y = {B ¯ ¯¯¯ ¯ X}_{K} c \in B_{¯ ¯ ¯ S}$ , we have:

	$∥ y - ¯ ¯ ¯ y ∥_{2}$	$= ∥ k \sum i = 1 c_{i} ({A X}_{K} - {B ¯ ¯¯¯ ¯ X}_{K})_{i} ∥_{2} \leq ε k \sum i = 1 \| c_{i} \|$
		$\leq ε \sqrt{k} ∥ c ∥_{2} \leq \frac{ε}{L_{k} ({A X}_{K})} ∥ y ∥_{2},$

where the last inequality follows directly from (5). From Def. 3:

d (A_{S}, B_{¯ ¯ ¯ S}) \leq \frac{}{ε} L_{k} ({A X}_{K}) \leq ε \frac{C_{1}}{C_{2}},

(18)

where the second inequality is due to $L_{k} ({A X}_{K}) \geq L_{k} ({A X}_{I (S)})$ and (15). Finally, apply Lem. 1 with $ε < L_{2} (A) / C_{1}$ . ∎

Proof of Thm. 2.

We bound the number of $k$ -sparse ${¯ ¯ ¯ x}_{i}$ and then apply Thm. 1. Let $n_{p}$ be the number of ${¯ ¯ ¯ x}_{i}$ with $∥ {¯ ¯ ¯ x}_{i} ∥_{0} = p$ . Since the $x_{i}$ are all $k$ -sparse, by (4) we have: $k \sum_{p = 0}^{¯ ¯¯ ¯ m} n_{p} \geq \sum_{i = 0}^{N} ∥ x_{i} ∥_{0} \geq \sum_{i = 0}^{N} ∥ {¯ ¯ ¯ x}_{i} ∥_{0} = \sum_{p = 0}^{¯ ¯¯ ¯ m} p n_{p} .$ Hence,

¯ ¯¯ ¯ m \sum p = k + 1 n_{p} \leq ¯ ¯¯ ¯ m \sum p = k + 1 (p - k) n_{p} \leq k \sum p = 0 (k - p) n_{p} \leq k k - 1 \sum p = 0 n_{p},

(19)

demonstrating that the number vectors ${¯ ¯ ¯ x}_{i}$ that are not $k$ -sparse is controlled by how many are $(k - 1)$ -sparse.

Next, observe that no more than $(k - 1) | H |$ of the ${¯ ¯ ¯ x}_{i}$ share a support $¯ ¯¯ ¯ S$ of size less than $k$ ; otherwise, by the pigeonhole principle, at least $k$ of these indices $i$ belong to the same $K \subseteq I (S)$ for some $S \in H$ and (as argued previously) (18) follows. Since the right-hand side of (18) is less than one, by (11) we have the contradiction $k = dim (A_{S}) \leq dim (B_{¯ ¯ ¯ S}) \leq | ¯ ¯¯ ¯ S | .$

The total number of $(k - 1)$ -sparse vectors ${¯ ¯ ¯ x}_{i}$ thus can not exceed $| H | (k - 1) (\frac{¯ ¯¯ ¯ m}{k - 1})$ . By (19), no more than $| H | k (k - 1) (\frac{¯ ¯¯ ¯ m}{k - 1})$ vectors ${¯ ¯ ¯ x}_{i}$ are not $k$ -sparse. Since for every $S \in H$ there are over $(k - 1) [(\frac{¯ ¯¯ ¯ m}{k}) + | H | k (\frac{¯ ¯¯ ¯ m}{k - 1})]$ vectors $x_{i}$ supported there, it must be that more than $(k - 1) (\frac{¯ ¯¯ ¯ m}{k})$ of them have corresponding ${¯ ¯ ¯ x}_{i}$ that are $k$ -sparse. The result now follows from Thm. 1. ∎

Proof (sketch) of Thm. 3.

Let M be the matrix with columns $A x_{i}$ , $i \in [N]$ . Consider the polynomial $\prod_{S \in (\frac{[N]}{2 k})} \sum_{S^{'} \in (\frac{[n]}{2 k})} (det M_{S^{'}, S})^{2}$ in the indeterminate entries of $A$ and $x_{i}$ , with notation as in Sec. II. It can be checked that when this polynomial is nonzero for a substitution of real numbers for the indeterminates, all of the genericity requirements on $A$ and $x_{i}$ in our proofs of stability in Thm. 1 are satisfied (in particular, the spark condition (2) on $A$ ). ∎

Proof (sketch) of Cor. 2.

First, note that if a set of measure spaces ${(X_{ℓ}, Σ_{ℓ}, ν_{ℓ})}_{ℓ = 1}^{p}$ has that $ν_{ℓ}$ is absolutely continuous with respect to $μ$ for all $ℓ \in [p]$ , where $μ$ is the standard Borel measure on $R$ , then the product measure $\prod_{ℓ = 1}^{p} ν_{ℓ}$ is absolutely continuous with respect to the standard Borel product measure on $R^{p}$ . By Thm. 3, there is a polynomial that is nonzero when $Y$ has a stable $k$ -sparse representation in $R^{m}$ ; in particular, stability holds almost surely. ∎

Iv Discussion

The goal of this work was to explain the emergence of characteristic representations from sparse linear coding models fit to natural data, despite the varied assumptions underlying the many algorithms in current use. To this end, we have taken an important step toward unifying the hundreds (if not thousands) of publications on the topic by demonstrating very general, deterministic conditions under which the identification of parameters in this parsimonious model is not only possible but also robust to the inevitable uncertainty permeating measurement and model choice.

Specifically, we have shown that, given sufficient data, the problem of seeking a dictionary and sparse codes with minimal average support size (Prob. 2) reduces to an instance of Prob. 1, to which our main result (Thm. 1) applies: every dictionary and sparse codes consistent with the data are equivalent up to inherent relabeling/scaling ambiguities and a discrepancy (error) that scales linearly with the measurement noise or modeling inaccuracy. The constants we provide are explicit and computable; as such, there is an effective procedure that sufficiently affirms if a proposed solution to Probs. 1 or 2 is indeed unique up to noise and inherent ambiguities.

An immediate application of our theoretical work is Prop. 1, which certifies the validity of perhaps the only published theory of neurally-plausible bottleneck communication in the brain: that sparse linear coding in a compressed space of neural activity can recover sparse codes sent through a randomly-constructed (but unknown) noisy wiring bottleneck [9].⁷⁷7We refer the reader to [29] for more on interactions between dictionary learning and neuroscience.

Beyond an original extension of existing noiseless guarantees [16] to the noisy regime and their application to Prob. 2, a major innovation in our work is a theory of combinatorial designs for support sets key to the identification of the dictionary. We incorporate this idea into a new fundamental lemma in matrix theory (Lem. 1) that draws upon the definition of a new matrix lower bound induced by a hypergraph. Insights enabled by our combinatorial approach include: 1) a subset of dictionary elements is recoverable even if dictionary size is overestimated, 2) data require only a polynomial number of distinct sparse supports, and 3) the spark condition is not a necessary property of recoverable dictionaries.

A technical difficulty in proving Thm. 1 is the absence of any assumption at all on dictionaries in solutions to Prob. 1. We sought such a guarantee because of the practical difficulty of ensuring that an algorithm maintain a dictionary satisfying the spark condition (2) at each iteration, an implicit requirement of all previous works except [16]; indeed, even certifying a dictionary has this property is NP-hard [20].

In fact, uniqueness guarantees with minimal assumptions apply to all areas of data science and engineering that utilize learned sparse structure. For example, several groups have applied compressed sensing to signal processing tasks: MRI analysis

[30], image compression [31], and, more recently, the design of an ultrafast camera [32]. Given such effective uses of compressed sensing, it is only a matter of time before these systems incorporate dictionary learning to encode and decode signals (e.g., in a device that learns structure from motion [33]) just as scientists have used it to make sense of their data [34, 35, 36, 37]. Assurances such as those offered by our theorems certify that different devices (with different initialization, etc.) will learn equivalent representations given enough data from statistically identical systems.⁸⁸8

To contrast with the current hot topic of “Deep Learning”, there are few such uniqueness guarantees for these models of data; moreover, even small noise can dramatically alter their output

[38]. Indeed, it seems a main reason for the sustained interest in dictionary learning as an unsupervised method for data analysis is the assumed well-posedness of parameter identification in the sparse linear coding model, confirmation of which forms the core of our theoretical findings.

Acknowledgment

We thank Fritz Sommer for turning our attention to the dictionary learning problem and Darren Rhea for sharing early explorations. We also thank Ian Morris for posting a reference to his proof of (12) online at Stack Exchange. Finally, we thank Bizzyskillet (www.soundcloud.com/bizzyskillet) for the “No Exam Jams”, which played on repeat during many long hours designing proofs.

References

[1] Y. Sato, “A method of self-recovering equalization for multilevel amplitude-modulation systems,” IEEE Trans. Commun., vol. 23, no. 6, pp. 679–682, 1975.
[2] Z. Zhang, Y. Xu, J. Yang, X. Li, and D. Zhang, “A survey of sparse representation: algorithms and applications,” Access, IEEE, vol. 3, pp. 490–530, 2015.
[3] Z. Wang, J. Yang, H. Zhang, Z. Wang, Y. Yang, D. Liu, and T. Huang, Sparse coding and its applications in computer vision
. World Scientific, 2015.
[4] B. Olshausen and D. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, vol. 381, no. 6583, pp. 607–609, 1996.
[5]
J. Hurri, A. Hyvärinen, J. Karhunen, and E. Oja, “Image feature extraction using independent component analysis,” in
Proc. NORSIG ’96 (Nordic Signal Proc. Symposium), 1996, pp. 475–478.
[6] A. Bell and T. Sejnowski, “The “independent components” of natural scenes are edge filters,” Vision Res., vol. 37, no. 23, pp. 3327–3338, 1997.
[7] J. van Hateren and A. van der Schaaf, “Independent component filters of natural images compared with simple cells in primary visual cortex,” Proc. R Soc. Lond. [Biol.], vol. 265, no. 1394, pp. 359–366, 1998.
[8] D. Donoho and A. Flesia, “Can recent innovations in harmonic analysis ’explain’ key findings in natural image statistics?” Network Comp. Neural, vol. 12, no. 3, pp. 371–393, 2001.
[9] G. Isely, C. Hillar, and F. Sommer, “Deciphering subsampled data: adaptive compressive sampling as a principle of brain communication,” in Adv. Neural Inf. Process. Syst., 2010, pp. 910–918.
[10] J. Hughes, D. Graham, and D. Rockmore, “Quantification of artistic style through sparse coding analysis in the drawings of Pieter Bruegel the Elder,” Proc. Natl. Acad. Sci., vol. 107, no. 4, pp. 1279–1283, 2010.
[11] J. Sun, Q. Qu, and J. Wright, “Complete dictionary recovery over the sphere I: Overview and the geometric picture,” IEEE Trans. Inf. Theory, pp. 853 – 884, 2016.
[12] J. Hadamard, “Sur les problèmes aux dérivées partielles et leur signification physique,” Princeton University Bulletin, vol. 13, no. 49-52, p. 28, 1902.
[13] Y. Li, A. Cichocki, and S.-I. Amari, “Analysis of sparse representation and blind source separation,” Neural Comput., vol. 16, no. 6, pp. 1193–1234, 2004.
[14] P. Georgiev, F. Theis, and A. Cichocki, “Sparse component analysis and blind source separation of underdetermined mixtures,” IEEE Trans. Neural Netw., vol. 16, pp. 992–996, 2005.
[15]

When is sparse dictionary learning well-posed?

Authors

Efficient and Parallel Separable Dictionary Learning

A Study on Unsupervised Dictionary Learning and Feature Encoding for Action Classification

Anomaly-Sensitive Dictionary Learning for Unsupervised Diagnostics of Solid Media

Sparse and spurious: dictionary learning with noise and outliers

Multimodal Sparse Bayesian Dictionary Learning

The Statistical Inefficiency of Sparse Coding for Images (or, One Gabor to Rule them All)

Boosting Dictionary Learning with Error Codes

I Introduction

Problem 1 (Sparse linear coding).

Definition 1.

Problem 2.

Ii Results

Definition 2.

Theorem 1.

Corollary 1.

Theorem 2.

Theorem 3.

Corollary 2.

Proposition 1.

Iii Proofs

Proof of Thm. 1 for $k = 1$ .

Definition 3.

Lemma 1.

Definition 4.

Proof of Thm. 1 for $k < m$ .

Proof of Thm. 2.

Proof (sketch) of Thm. 3.

Proof (sketch) of Cor. 2.

Iv Discussion

Acknowledgment

References

When is sparse dictionary learning well-posed?

Authors

Efficient and Parallel Separable Dictionary Learning

A Study on Unsupervised Dictionary Learning and Feature Encoding for Action Classification

Anomaly-Sensitive Dictionary Learning for Unsupervised Diagnostics of Solid Media

Sparse and spurious: dictionary learning with noise and outliers

Multimodal Sparse Bayesian Dictionary Learning

The Statistical Inefficiency of Sparse Coding for Images (or, One Gabor to Rule them All)

Boosting Dictionary Learning with Error Codes

This week in AI

I Introduction

Problem 1 (Sparse linear coding).

Definition 1.

Problem 2.

Ii Results

Definition 2.

Theorem 1.

Corollary 1.

Theorem 2.

Theorem 3.

Corollary 2.

Proposition 1.

Iii Proofs

Proof of Thm. 1 for k=1.

Definition 3.

Lemma 1.

Definition 4.

Proof of Thm. 1 for k<m.

Proof of Thm. 2.

Proof (sketch) of Thm. 3.

Proof (sketch) of Cor. 2.

Iv Discussion

Acknowledgment

References

Proof of Thm. 1 for $k = 1$ .

Proof of Thm. 1 for $k < m$ .