Adaptive Sparse Structure Development with Pruning and Regeneration for Spiking Neural Networks

11/22/2022
by   Bing Han, et al.
0

Spiking Neural Networks (SNNs) are more biologically plausible and computationally efficient. Therefore, SNNs have the natural advantage of drawing the sparse structural plasticity of brain development to alleviate the energy problems of deep neural networks caused by their complex and fixed structures. However, previous SNNs compression works are lack of in-depth inspiration from the brain development plasticity mechanism. This paper proposed a novel method for the adaptive structural development of SNN (SD-SNN), introducing dendritic spine plasticity-based synaptic constraint, neuronal pruning and synaptic regeneration. We found that synaptic constraint and neuronal pruning can detect and remove a large amount of redundancy in SNNs, coupled with synaptic regeneration can effectively prevent and repair over-pruning. Moreover, inspired by the neurotrophic hypothesis, neuronal pruning rate and synaptic regeneration rate were adaptively adjusted during the learning-while-pruning process, which eventually led to the structural stability of SNNs. Experimental results on spatial (MNIST, CIFAR-10) and temporal neuromorphic (N-MNIST, DVS-Gesture) datasets demonstrate that our method can flexibly learn appropriate compression rate for various tasks and effectively achieve superior performance while massively reducing the network energy consumption. Specifically, for the spatial MNIST dataset, our SD-SNN achieves 99.51% accuracy at the pruning rate 49.83%, which has a 0.05% accuracy improvement compared to the baseline without compression. For the neuromorphic DVS-Gesture dataset, 98.20% accuracy with 1.09% improvement is achieved by our method when the compression rate reaches 55.50%.

READ FULL TEXT

page 1

page 8

research
05/11/2021

Pruning of Deep Spiking Neural Networks through Gradient Rewiring

Spiking Neural Networks (SNNs) have been attached great importance due t...
research
11/23/2022

Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks

Developmental plasticity plays a prominent role in shaping the brain's s...
research
06/06/2023

ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

Spiking neural networks (SNNs) have manifested remarkable advantages in ...
research
04/19/2023

Biologically inspired structure learning with reverse knowledge distillation for spiking neural networks

Spiking neural networks (SNNs) have superb characteristics in sensory in...
research
10/11/2022

STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks

Spiking Neural Networks (SNNs), as one of the algorithmic models in neur...
research
08/04/2021

Composing Recurrent Spiking Neural Networks using Locally-Recurrent Motifs and Risk-Mitigating Architectural Optimization

In neural circuits, recurrent connectivity plays a crucial role in netwo...
research
09/22/2016

Regularized Dynamic Boltzmann Machine with Delay Pruning for Unsupervised Learning of Temporal Sequences

We introduce Delay Pruning, a simple yet powerful technique to regulariz...

Please sign up or login with your details

Forgot password? Click here to reset