Log In Sign Up

Dense and Sparse Coding: Theory and Architectures

by   Abiy Tasissa, et al.

The sparse representation model has been successfully utilized in a number of signal and image processing tasks; however, recent research has highlighted its limitations in certain deep-learning architectures. This paper proposes a novel dense and sparse coding model that considers the problem of recovering a dense vector 𝐱 and a sparse vector 𝐮 given linear measurements of the form 𝐲 = 𝐀𝐱+𝐁𝐮. Our first theoretical result proposes a new natural geometric condition based on the minimal angle between subspaces corresponding to the measurement matrices 𝐀 and 𝐁 to establish the uniqueness of solutions to the linear system. The second analysis shows that, under mild assumptions and sufficient linear measurements, a convex program recovers the dense and sparse components with high probability. The standard RIPless analysis cannot be directly applied to this setup. Our proof is a non-trivial adaptation of techniques from anisotropic compressive sensing theory and is based on an analysis of a matrix derived from the measurement matrices 𝐀 and 𝐁. We begin by demonstrating the effectiveness of the proposed model on simulated data. Then, to address its use in a dictionary learning setting, we propose a dense and sparse auto-encoder (DenSaE) that is tailored to it. We demonstrate that a) DenSaE denoises natural images better than architectures derived from the sparse coding model (𝐁𝐮), b) training the biases in the latter amounts to implicitly learning the 𝐀𝐱 + 𝐁𝐮 model, and c) 𝐀 and 𝐁 capture low- and high-frequency contents, respectively.


page 5

page 6

page 13

page 14

page 15


Group-based Sparse Representation for Image Restoration

Traditional patch-based sparse representation modeling of natural images...

Riemannian Dictionary Learning and Sparse Coding for Positive Definite Matrices

Data encoded as symmetric positive definite (SPD) matrices frequently ar...

A Unified Recovery of Structured Signals Using Atomic Norm

In many applications we seek to recover signals from linear measurements...

Forest Sparsity for Multi-channel Compressive Sensing

In this paper, we investigate a new compressive sensing model for multi-...

Deeply-Sparse Signal rePresentations (DS^2P)

The solution to the regularized least-squares problem min_x∈R^p+1/2y-Ax_...

Convolutional Dictionary Learning in Hierarchical Networks

Filter banks are a popular tool for the analysis of piecewise smooth sig...

Graph Regularized Tensor Sparse Coding for Image Representation

Sparse coding (SC) is an unsupervised learning scheme that has received ...