Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks

05/24/2020
by   Bojian Yin, et al.
20

The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a >100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks.

READ FULL TEXT

page 7

page 8

research
03/12/2021

Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

Inspired by more detailed modeling of biological neurons, Spiking neural...
research
06/06/2023

Spike-based computation using classical recurrent neural networks

Spiking neural networks are a type of artificial neural networks in whic...
research
06/22/2023

Accelerating SNN Training with Stochastic Parallelizable Spiking Neurons

Spiking neural networks (SNN) are able to learn spatiotemporal features ...
research
12/20/2021

Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time

The event-driven and sparse nature of communication between spiking neur...
research
06/13/2022

EGRU: Event-based GRU for activity-sparse inference and learning

The scalability of recurrent neural networks (RNNs) is hindered by the s...
research
09/17/2019

Control of criticality and computation in spiking neuromorphic networks with plasticity

The critical state is assumed to be optimal for any computation in recur...
research
12/17/2018

Deep Networks Incorporating Spiking Neural Dynamics

Neural networks have become the key technology of Artificial Intelligenc...

Please sign up or login with your details

Forgot password? Click here to reset