Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition

12/01/2022
by   Alexandre Bittar, et al.
0

Compared to conventional artificial neurons that produce dense and real-valued responses, biologically-inspired spiking neurons transmit sparse and binary information, which can also lead to energy-efficient implementations. Recent research has shown that spiking neural networks can be trained like standard recurrent neural networks using the surrogate gradient method. They have shown promising results on speech command recognition tasks. Using the same technique, we show that they are scalable to large vocabulary continuous speech recognition, where they are capable of replacing LSTMs in the encoder with only minor loss of performance. This suggests that they may be applicable to more involved sequence-to-sequence tasks. Moreover, in contrast to their recurrent non-spiking counterparts, they show robustness to exploding gradient problems without the need to use gates.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/12/2021

Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

Inspired by more detailed modeling of biological neurons, Spiking neural...
research
11/13/2020

Low-activity supervised convolutional spiking neural networks applied to speech commands recognition

Deep Neural Networks (DNNs) are the current state-of-the-art models in m...
research
12/13/2021

Improving Surrogate Gradient Learning in Spiking Neural Networks via Regularization and Normalization

Spiking neural networks (SNNs) are different from the classical networks...
research
11/19/2019

Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition

Artificial neural networks (ANN) have become the mainstream acoustic mod...
research
09/04/2021

Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning

Spiking neural networks (SNNs) with leaky integrate and fire (LIF) neuro...
research
04/26/2023

Membrane Potential Distribution Adjustment and Parametric Surrogate Gradient in Spiking Neural Networks

As an emerging network model, spiking neural networks (SNNs) have arouse...
research
10/06/2021

Spike-inspired Rank Coding for Fast and Accurate Recurrent Neural Networks

Biological spiking neural networks (SNNs) can temporally encode informat...

Please sign up or login with your details

Forgot password? Click here to reset