Subgradient Descent Learns Orthogonal Dictionaries

10/25/2018
by   Yu Bai, et al.
0

This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem. We show that a subgradient descent algorithm, with random initialization, can provably recover orthogonal dictionaries on a natural nonsmooth, nonconvex ℓ_1 minimization formulation of the problem, under mild statistical assumptions on the data. This is in contrast to previous provable methods that require either expensive computation or delicate initialization schemes. Our analysis develops several tools for characterizing landscapes of nonsmooth functions, which might be of independent interest for provable training of deep networks with nonsmooth activations (e.g., ReLU), among numerous other applications. Preliminary experiments corroborate our analysis and show that our algorithm works well empirically in recovering orthogonal dictionaries.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/09/2017

Alternating minimization for dictionary learning with random initialization

We present theoretical guarantees for an alternating minimization algori...
research
08/28/2013

New Algorithms for Learning Incoherent and Overcomplete Dictionaries

In sparse recovery we are given a matrix A (the dictionary) and a vector...
research
01/03/2014

More Algorithms for Provable Dictionary Learning

In dictionary learning, also known as sparse coding, the algorithm is gi...
research
04/24/2018

On Learning Sparsely Used Dictionaries from Incomplete Samples

Most existing algorithms for dictionary learning assume that all entries...
research
10/24/2019

Community-Level Anomaly Detection for Anti-Money Laundering

Anomaly detection in networks often boils down to identifying an underly...
research
08/12/2017

Sparse Coding and Autoencoders

In "Dictionary Learning" one tries to recover incoherent matrices A^* ∈R...
research
04/23/2018

Towards Learning Sparsely Used Dictionaries with Arbitrary Supports

Dictionary learning is a popular approach for inferring a hidden basis o...

Please sign up or login with your details

Forgot password? Click here to reset