Understanding the Learned Iterative Soft Thresholding Algorithm with matrix factorization

06/02/2017
by   Thomas Moreau, et al.
0

Sparse coding is a core building block in many data analysis and machine learning pipelines. Typically it is solved by relying on generic optimization techniques, such as the Iterative Soft Thresholding Algorithm and its accelerated version (ISTA, FISTA). These methods are optimal in the class of first-order methods for non-smooth, convex functions. However, they do not exploit the particular structure of the problem at hand nor the input data distribution. An acceleration using neural networks, coined LISTA, was proposed in Gregor and Le Cun (2010), which showed empirically that one could achieve high quality estimates with few iterations by modifying the parameters of the proximal splitting appropriately. In this paper we study the reasons for such acceleration. Our mathematical analysis reveals that it is related to a specific matrix factorization of the Gram kernel of the dictionary, which attempts to nearly diagonalise the kernel with a basis that produces a small perturbation of the ℓ_1 ball. When this factorization succeeds, we prove that the resulting splitting algorithm enjoys an improved convergence bound with respect to the non-adaptive version. Moreover, our analysis also shows that conditions for acceleration occur mostly at the beginning of the iterative process, consistent with numerical experiments. We further validate our analysis by showing that on dictionaries where this factorization does not exist, adaptive acceleration fails.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 15

09/01/2016

Understanding Trainable Sparse Coding via Matrix Factorization

Sparse coding is a core building block in many data analysis and machine...
07/09/2021

Block Alternating Bregman Majorization Minimization with Extrapolation

In this paper, we consider a class of nonsmooth nonconvex optimization p...
11/02/2021

Learning Multiresolution Matrix Factorization and its Wavelet Networks on Graphs

Multiresolution Matrix Factorization (MMF) is unusual amongst fast matri...
05/27/2019

Learning step sizes for unfolded sparse coding

Sparse coding is typically solved by iterative optimization techniques, ...
03/16/2017

Accelerated and Inexact Soft-Impute for Large-Scale Matrix and Tensor Completion

Matrix and tensor completion aim to recover a low-rank matrix / tensor f...
05/16/2017

The Incremental Multiresolution Matrix Factorization Algorithm

Multiresolution analysis and matrix factorization are foundational tools...
01/23/2020

Ada-LISTA: Learned Solvers Adaptive to Varying Models

Neural networks that are based on unfolding of an iterative solver, such...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.