Advancing Deep Residual Learning by Solving the Crux of Degradation in Spiking Neural Networks

12/09/2021
by   Yifan Hu, et al.
0

Despite the rapid progress of neuromorphic computing, the inadequate depth and the resulting insufficient representation power of spiking neural networks (SNNs) severely restrict their application scope in practice. Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks, but rarely did previous work assess their applicability to the characteristics of spike-based communication and spatiotemporal dynamics. This negligence leads to impeded information flow and the accompanying degradation problem. In this paper, we identify the crux and then propose a novel residual block for SNNs, which is able to significantly extend the depth of directly trained SNNs, e.g., up to 482 layers on CIFAR-10 and 104 layers on ImageNet, without observing any slight degradation problem. We validate the effectiveness of our methods on both frame-based and neuromorphic datasets, and our SRM-ResNet104 achieves a superior result of 76.02 first time in the domain of directly trained SNNs. The great energy efficiency is estimated and the resulting networks need on average only one spike per neuron for classifying an input sample. We believe our powerful and scalable modeling will provide a strong support for further exploration of SNNs.

READ FULL TEXT
research
12/15/2021

Advancing Residual Learning towards Powerful Deep Spiking Neural Networks

Despite the rapid progress of neuromorphic computing, inadequate capacit...
research
02/08/2021

Spike-based Residual Blocks

Deep Spiking Neural Networks (SNNs) are harder to train than ANNs becaus...
research
10/12/2022

Multi-Level Firing with Spiking DS-ResNet: Enabling Better and Deeper Directly-Trained Spiking Neural Networks

Spiking neural networks (SNNs) are bio-inspired neural networks with asy...
research
07/16/2023

Enhancing Energy Efficiency and Reliability in Autonomous Systems Estimation using Neuromorphic Approach

Energy efficiency and reliability have long been crucial factors for ens...
research
07/13/2020

Coarse scale representation of spiking neural networks: backpropagation through spikes and application to neuromorphic hardware

In this work we explore recurrent representations of leaky integrate and...
research
03/12/2019

Spiking-YOLO: Spiking Neural Network for Real-time Object Detection

Over the past decade, deep neural networks (DNNs) have become a de-facto...
research
09/08/2020

Nonlinear computations in spiking neural networks through multiplicative synapses

The brain performs many nonlinear computations through intricate spiking...

Please sign up or login with your details

Forgot password? Click here to reset