Spatially-Coupled Neural Network Architectures

by   Arman Hasanzadeh, et al.

In this work, we leverage advances in sparse coding techniques to reduce the number of trainable parameters in a fully connected neural network. While most of the works in literature impose ℓ_1 regularization, DropOut or DropConnect techniques to induce sparsity, our scheme considers feature importance as a criterion to allocate the trainable parameters (resources) efficiently in the network. Even though sparsity is ensured, ℓ_1 regularization requires training on all the resources in a deep neural network. The DropOut/DropConnect techniques reduce the number of trainable parameters in the training stage by dropping a random collection of neurons/edges in the hidden layers. However, both these techniques do not pay heed to the underlying structure in the data when dropping the neurons/edges. Moreover, these frameworks require a storage space equivalent to the number of parameters in a fully connected neural network. We address the above issues with a more structured architecture inspired from spatially-coupled sparse constructions. The proposed architecture is shown to have a performance akin to a conventional fully connected neural network with dropouts, and yet achieving a 94% reduction in the training parameters. Extensive simulations are presented and the performance of the proposed scheme is compared against traditional neural network architectures.


page 1

page 2

page 3

page 4


Sparse Neural Networks Topologies

We propose Sparse Neural Network architectures that are based on random ...

Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performance

Current research suggests that the key factors in designing neural netwo...

Reduction of Overfitting in Diabetes Prediction Using Deep Learning Neural Network

Augmented accuracy in prediction of diabetes will open up new frontiers ...

An exploration of parameter redundancy in deep networks with circulant projections

We explore the redundancy of parameters in deep neural networks by repla...

Layerwise Sparsifying Training and Sequential Learning Strategy for Neural Architecture Adaptation

This work presents a two-stage framework for progressively developing ne...

Discriminative convolutional Fisher vector network for action recognition

In this work we propose a novel neural network architecture for the prob...

Connecting Sphere Manifolds Hierarchically for Regularization

This paper considers classification problems with hierarchically organiz...

Please sign up or login with your details

Forgot password? Click here to reset