Spatially-Coupled Neural Network Architectures

07/03/2019
by   Arman Hasanzadeh, et al.
0

In this work, we leverage advances in sparse coding techniques to reduce the number of trainable parameters in a fully connected neural network. While most of the works in literature impose ℓ_1 regularization, DropOut or DropConnect techniques to induce sparsity, our scheme considers feature importance as a criterion to allocate the trainable parameters (resources) efficiently in the network. Even though sparsity is ensured, ℓ_1 regularization requires training on all the resources in a deep neural network. The DropOut/DropConnect techniques reduce the number of trainable parameters in the training stage by dropping a random collection of neurons/edges in the hidden layers. However, both these techniques do not pay heed to the underlying structure in the data when dropping the neurons/edges. Moreover, these frameworks require a storage space equivalent to the number of parameters in a fully connected neural network. We address the above issues with a more structured architecture inspired from spatially-coupled sparse constructions. The proposed architecture is shown to have a performance akin to a conventional fully connected neural network with dropouts, and yet achieving a 94% reduction in the training parameters. Extensive simulations are presented and the performance of the proposed scheme is compared against traditional neural network architectures.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/18/2017

Sparse Neural Networks Topologies

We propose Sparse Neural Network architectures that are based on random ...
research
09/16/2020

Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performance

Current research suggests that the key factors in designing neural netwo...
research
07/26/2017

Reduction of Overfitting in Diabetes Prediction Using Deep Learning Neural Network

Augmented accuracy in prediction of diabetes will open up new frontiers ...
research
02/11/2015

An exploration of parameter redundancy in deep networks with circulant projections

We explore the redundancy of parameters in deep neural networks by repla...
research
11/13/2022

Layerwise Sparsifying Training and Sequential Learning Strategy for Neural Architecture Adaptation

This work presents a two-stage framework for progressively developing ne...
research
07/19/2017

Discriminative convolutional Fisher vector network for action recognition

In this work we propose a novel neural network architecture for the prob...
research
06/25/2021

Connecting Sphere Manifolds Hierarchically for Regularization

This paper considers classification problems with hierarchically organiz...

Please sign up or login with your details

Forgot password? Click here to reset