Sparse Spiking Gradient Descent

05/18/2021
by   Nicolas Perez Nieves, et al.
10

There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods while being significantly faster and more memory efficient. We show the effectiveness of our method on real datasets of varying complexity (Fashion-MNIST, Neuromophic-MNIST and Spiking Heidelberg Digits) achieving a speedup in the backward pass of up to 70x, and 40 without losing accuracy.

READ FULL TEXT

page 7

page 9

page 16

page 20

page 21

page 23

05/30/2022

Accelerating spiking neural network training

Spiking neural networks (SNN) are a type of artificial network inspired ...
12/03/2019

Optimizing the energy consumption of spiking neural networks for neuromorphic applications

In the last few years, spiking neural networks have been demonstrated to...
06/13/2022

Spiking Neural Networks for Frame-based and Event-based Single Object Localization

Spiking neural networks have shown much promise as an energy-efficient a...
06/12/2020

Training spiking multi-layer networks with surrogate gradients on an analog neuromorphic substrate

Spiking neural networks are nature's solution for parallel information p...
06/03/2020

You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy

In the past decade, advances in Artificial Neural Networks (ANNs) have a...
05/25/2019

A neuromorphic boost to RNNs using low pass filters

The increasing difficulty with Moore's law scaling and the remarkable su...
10/04/2012

A network of spiking neurons for computing sparse representations in an energy efficient way

Computing sparse redundant representations is an important problem both ...