Effect of top-down connections in Hierarchical Sparse Coding

by   Victor Boutin, et al.

Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images. The simplest solution to solve this computationally hard problem is to decompose it into independent layer-wise subproblems. However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers. In this study, a new model called 2-Layers Sparse Predictive Coding (2L-SPC) is introduced to assess the impact of this inter-layer feedback connection. In particular, the 2L-SPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of independent Lasso layers. The 2L-SPC and the 2-layers Hi-La networks are trained on 4 different databases and with different sparsity parameters on each layer. First, we show that the overall prediction error generated by 2L-SPC is lower thanks to the feedback mechanism as it transfers prediction error between layers. Second, we demonstrate that the inference stage of the 2L-SPC is faster to converge than for the Hi-La model. Third, we show that the 2L-SPC also accelerates the learning process. Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the 2L-SPC features are more generic and informative.


page 19

page 29

page 30

page 31

page 32

page 33


Meaningful representations emerge from Sparse Deep Predictive Coding

Convolutional Neural Networks (CNNs) are the state-of-the-art algorithms...

Deep Predictive Coding Network for Object Recognition

Inspired by predictive coding in neuroscience, we designed a bi-directio...

Habitual and Reflective Control in Hierarchical Predictive Coding

In cognitive science, behaviour is often separated into two types. Refle...

AFA-PredNet: The action modulation within predictive coding

The predictive processing (PP) hypothesizes that the predictive inferenc...

Inference via Sparse Coding in a Hierarchical Vision Model

Sparse coding has been incorporated in models of the visual cortex for i...

Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition

Inspired by "predictive coding" - a theory in neuroscience, we develop a...

Preventing Deterioration of Classification Accuracy in Predictive Coding Networks

Predictive Coding Networks (PCNs) aim to learn a generative model of the...

Please sign up or login with your details

Forgot password? Click here to reset