Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life

by   Tim Whitaker, et al.

Rectified Linear Units (ReLU) are the default choice for activation functions in deep neural networks. While they demonstrate excellent empirical performance, ReLU activations can fall victim to the dead neuron problem. In these cases, the weights feeding into a neuron end up being pushed into a state where the neuron outputs zero for all inputs. Consequently, the gradient is also zero for all inputs, which means that the weights which feed into the neuron cannot update. The neuron is not able to recover from direct back propagation and model capacity is reduced as those parameters can no longer be further optimized. Inspired by a neurological process of the same name, we introduce Synaptic Stripping as a means to combat this dead neuron problem. By automatically removing problematic connections during training, we can regenerate dead neurons and significantly improve model capacity and parametric utilization. Synaptic Stripping is easy to implement and results in sparse networks that are more efficient than the dense networks they are derived from. We conduct several ablation studies to investigate these dynamics as a function of network width and depth and we conduct an exploration of Synaptic Stripping with Vision Transformers on a variety of benchmark datasets.


Learning Neural Activations

An artificial neuron is modelled as a weighted summation followed by an ...

SynapCountJ --- a Tool for Analyzing Synaptic Densities in Neurons

The quantification of synapses is instrumental to measure the evolution ...

Enabling Resource-Aware Mapping of Spiking Neural Networks via Spatial Decomposition

With growing model complexity, mapping Spiking Neural Network (SNN)-base...

Learning with Delayed Synaptic Plasticity

The plasticity property of biological neural networks allows them to per...

Channel Modeling for Synaptic Molecular Communication With Re-uptake and Reversible Receptor Binding

In Diffusive Molecular Communication (DMC), information is transmitted b...

Solving internal covariate shift in deep learning with linked neurons

This work proposes a novel solution to the problem of internal covariate...

Please sign up or login with your details

Forgot password? Click here to reset