Dense and Sparse Coding: Theory and Architectures

06/16/2020 ∙ by Abiy Tasissa, et al. ∙ 0

The sparse representation model has been successfully utilized in a number of signal and image processing tasks; however, recent research has highlighted its limitations in certain deep-learning architectures. This paper proposes a novel dense and sparse coding model that considers the problem of recovering a dense vector 𝐱 and a sparse vector 𝐮 given linear measurements of the form 𝐲 = 𝐀𝐱+𝐁𝐮. Our first theoretical result proposes a new natural geometric condition based on the minimal angle between subspaces corresponding to the measurement matrices 𝐀 and 𝐁 to establish the uniqueness of solutions to the linear system. The second analysis shows that, under mild assumptions and sufficient linear measurements, a convex program recovers the dense and sparse components with high probability. The standard RIPless analysis cannot be directly applied to this setup. Our proof is a non-trivial adaptation of techniques from anisotropic compressive sensing theory and is based on an analysis of a matrix derived from the measurement matrices 𝐀 and 𝐁. We begin by demonstrating the effectiveness of the proposed model on simulated data. Then, to address its use in a dictionary learning setting, we propose a dense and sparse auto-encoder (DenSaE) that is tailored to it. We demonstrate that a) DenSaE denoises natural images better than architectures derived from the sparse coding model (𝐁𝐮), b) training the biases in the latter amounts to implicitly learning the 𝐀𝐱 + 𝐁𝐮 model, and c) 𝐀 and 𝐁 capture low- and high-frequency contents, respectively.



There are no comments yet.


page 5

page 6

page 13

page 14

page 15

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.