Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition

11/19/2019 ∙ by Jibin Wu, et al. ∙ 24

Artificial neural networks (ANN) have become the mainstream acoustic modeling technique for large vocabulary automatic speech recognition (ASR). A conventional ANN features a multi-layer architecture that requires massive amounts of computation. The brain-inspired spiking neural networks (SNN) closely mimic the biological neural networks and can operate on low-power neuromorphic hardware with spike-based computation. Motivated by their unprecedented energyefficiency and rapid information processing capability, we explore the use of SNNs for speech recognition. In this work, we use SNNs for acoustic modeling and evaluate their performance on several large vocabulary recognition scenarios. The experimental results demonstrate competitive ASR accuracies to their ANN counterparts, while require significantly reduced computational cost and inference time. Integrating the algorithmic power of deep SNNs with energy-efficient neuromorphic hardware, therefore, offer an attractive solution for ASR applications running locally on mobile and embedded devices.



There are no comments yet.


page 6

page 16

page 17

page 18

page 20

page 21

page 22

page 23

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


Artificial neural networks (ANN) have become the mainstream acoustic modeling technique for large vocabulary automatic speech recognition (ASR). A conventional ANN features a multi-layer architecture that requires massive amounts of computation. The brain-inspired spiking neural networks (SNN) closely mimic the biological neural networks and can operate on low-power neuromorphic hardware with spike-based computation. Motivated by their unprecedented energy-efficiency and rapid information processing capability, we explore the use of SNNs for speech recognition. In this work, we use SNNs for acoustic modeling and evaluate their performance on several large vocabulary recognition scenarios. The experimental results demonstrate competitive ASR accuracies to their ANN counterparts, while require significantly reduced computational cost and inference time. Integrating the algorithmic power of deep SNNs with energy-efficient neuromorphic hardware, therefore, offer an attractive solution for ASR applications running locally on mobile and embedded devices.

2 Keywords:

Deep Spiking Neural Networks, Automatic Speech Recognition, Tandem Learning, Neuromorphic Computing, Acoustic Modeling

2 Keywords:

Deep Spiking Neural Networks, Automatic Speech Recognition, Tandem Learning, Neuromorphic Computing, Acoustic Modeling

3 Introduction

Automatic speech recognition (ASR) has enabled the voice interface of mobile devices and smart home appliances in our everyday life. The rapid progress in the integration of voice interfaces has been viable on account of the remarkable performance of the ASR systems using artificial neural networks (ANN) for acoustic modeling (Lippmann, 1989; Lang et al., 1990; Hinton et al., 2012; Yu and Deng, 2015). Various ANN architectures, either feedforward or recurrent, have been investigated for modeling the acoustic information preserved in speech signals (Dahl et al., 2012; Graves et al., 2013; Abdel-Hamid et al., 2014).

The performance gains come with immense computational requirements often due to the time-synchronous processing of input audio signals. Several techniques have been proposed to reduce the computational load and memory storage of ANNs by reducing the number of parameters that have to be used for inference (Sainath et al., 2013; Xue et al., 2013; He et al., 2014; Povey et al., 2018). Another common solution, for reducing the amount of processed speech, uses a wake word or phrase to initialize the embedded ASR engine and starts listening to input speech (Zehetner et al., 2014; Sainath and Parada, 2015; Wu et al., 2018). Moreover, most devices with voice control rely on cloud-based ASR engines rather than local on-device solutions. The necessity of online processing of speech via cloud computing comes with various concerns, such as data security and processing speed, etc. There have been multiple efforts to develop on-device ASR solutions in which the speech signal is processed locally using the computational resources of mobile devices (Lei et al., 2013; McGraw et al., 2016).

Alternatively, event-driven models such as spiking neural networks (SNNs) inspired by the human brain have attracted ever-growing attention in recent years. The human brain is remarkably efficient and capable of performing complex perceptual and cognitive tasks. Notably, the adult’s brain only consumes about 20 watts to solve complex tasks that are equivalent to the power consumption of a dim light bulb (Laughlin and Sejnowski, 2003). While brain-inspired ANNs have demonstrated great capabilities in many perceptual (He et al., 2016; Xiong et al., 2017) and cognitive tasks (Silver et al., 2017)

, these models are computationally intensive and memory inefficient to operate as compared to the biological brains. Unlike ANNs, asynchronous and event-driven information processing of SNNs resembles the computing paradigm that observed in the human brains, whereby the energy consumption matches the activity levels of sensory stimuli. Given temporally sparse information transmitted in the surrounding environment, the event-driven computation, therefore, exhibits great computational efficiency than the synchronous computation used in ANNs.

Neuromorphic computing (NC), as a non-von Neumann computing paradigm, mimics the event-driven computation of the biological neural systems with SNN in silicon. The emerging neuromorphic computing architectures (Davies et al., 2018; Merolla et al., 2014; Furber et al., 2012) leverage on the massively parallel, low-power computing units to support spike-based information processing. Notably, the design of co-located memory and computing units effectively circumvents the von Neumann bottleneck of low-bandwidth between memory and the processing units (Monroe, 2014)

. Therefore, integrating the algorithmic power of deep SNNs with the compelling energy efficiency of NC hardware represents an intriguing solution for pervasive machine learning tasks and always-on applications. Furthermore, growing research efforts are devoted to developing novel non-volatile memory devices for ultra-low-power implementation of biological synapses and neurons

(Tang et al., 2019).

Some preliminary work on SNN-based phone classification or small-vocabulary speech recognition systems have been explored in (Jim-Shih Liaw and Berger, 1998; Näger et al., 2002; Loiselle et al., 2005; Holmberg et al., 2005; Kröger et al., 2009; Tavanaei and Maida, 2017a, b; Wu et al., 2018a, b; Zhang et al., 2015, 2019; Bellec et al., 2018). However, these SNN-based ASR systems are far from the scale and complexity of modern commercialized ANN-based ASR systems. It is mainly due to lacking effective training algorithms for deep SNNs and efficient software toolbox for SNN-based ASR systems.

Due to the discrete and non-differentiable nature of spike generation, the powerful error back-propagation algorithm is not directly applicable to the training of deep SNNs. Recently, considerable research efforts are devoted to addressing this problem and the resulting learning rules can be broadly categorized into the SNN-to-ANN conversion (Cao et al., 2015; Diehl et al., 2015), back-propagation through time with surrogate gradient (Neftci et al., 2019; Wu et al., 2018) and tandem learning (Wu et al., 2019). Despite several successful attempts on the large-scale image classification tasks with deep SNNs (Rueckauer et al., 2017; Hu et al., 2018; Sengupta et al., 2019; Wu et al., 2019), their applications to the large-vocabulary continuous ASR (LVCSR) tasks remain unexplored. In this work, we explore an SNN-based acoustic model for LVCSR using a recently proposed tandem learning rule (Wu et al., 2019) that supports an efficient and rapid inference. The experiments on several large vocabulary ASR benchmarks have shown competitive classification accuracies over the baseline ANN models. To the best of our knowledge, this is the first work that successfully applied SNNs for LVCSR tasks. Our preliminary study of an SNN-based acoustic model has also revealed compelling prospects of rapid inference and unprecedented energy efficiency of a neuromorphic approach.

The rest of the paper is organized as follows: In Section 4, we first give an overview of spiking neural networks, large vocabulary ASR systems, and existing SNN-based ASR systems. In Section 5, we introduce the spiking neuron model and the neural coding scheme that converts acoustic features into spike-based representation. We further present a recently introduced tandem learning framework for SNN training and how it is used to train deep SNN-based acoustic models. In Section 6, we present experimental results on the learning capability and energy efficiency of SNN-based acoustic models across three different types of recognition tasks including phone recognition, low-resourced and standard large-vocabulary ASR, and compare those to the ANN-based implementations. Finally, a discussion on the experimental findings is given in Section 7.

4 Fundamentals and Related Work

4.1 Spiking Neural Networks

The third generation spiking neural networks are originally studied as models to describe the information processing in the biological neural networks, wherein the information is communicated and exchanged via stereotypical action potentials or spikes (Gerstner and Kistler, 2002). Neuroscience studies reveal that the temporal structure and frequency of these spike trains are both important information carriers in the biological neural networks. As will be introduced in Section 5.1, the spiking neuron operates asynchronously and integrates the synaptic current from its incoming spike trains. An output spike is generated from the spiking neuron whenever its membrane potential crosses the firing threshold, and this output spike will be propagated to the connected neurons via the axon.

Motivated by the same connectionism principle, SNNs share the same network architectures, either feedforward or recurrent, with the conventional ANNs that use analog neurons. As shown in Figure 1, the early classification decision can be made from the SNN since the generation of the first output spike. However, the quality of the classification decision is typically improved over time with more evidence accumulated. It differs significantly from the synchronous information processing of the conventional ANNs, where the output layer needs to wait until all preceding layers are fully updated. Therefore, despite information is transmitted and processed at a speed that is several orders of magnitude slower in neural substrates than signal processing in modern transistors, biological neural systems can perform complex tasks rapidly. For more overviews about SNNs and their applications, we refer readers to (Pfeiffer and Pfeil, 2018; Tavanaei et al., 2018).

4.2 Large Vocabulary Automatic Speech Recognition

As shown in Figure 2

, conventional ASR systems uses acoustic and linguistic information preserved in three distinct components to convert speech signals to the corresponding text: (1) an acoustic model for preserving the statistical representations of different speech units, e.g. phones, from speech features, (2) a language model for assigning probabilities to the co-occurring word sequences and (3) a pronunciation lexicon for mapping the phonetic transcriptions to orthography. These resources are jointly used to determine the most likely hypothesis in the decoding stage.

The acoustic modeling has been achieved using various statistical models such as Gaussian Mixture Models (GMM) for assigning frame-level phone posteriors in conjunction with a Hidden Markov Model (HMM) for duration modeling 

(Yu and Deng, 2015). More recently, ANN-based approaches have become the standard acoustic models providing state-of-the-art performance across a wide spectrum of ASR tasks (Hinton et al., 2012). Together with the numerous ANN architectures explored for acoustic modeling, several end-to-end ANN architectures have been proposed for directly mapping speech features to text with optional use of the other linguistic components (Graves and Jaitly, 2014a; Chan et al., 2016; Watanabe et al., 2017).

The probabilistic definition of acoustic modeling becomes more evident via the Bayesian formulation of the speech recognition task. Given a target speech signal that segmented into T overlapped frames, the resulting frame-wise features can be represented as . An ASR system assigns the probability to all possible word sequences , and the word sequence with the highest probability is the recognized output,


The probability can be decomposed into two parts by applying the Bayes’ rule as below,


can be omitted as it does not depend on W. This results in


which formally defines the theoretical foundation that are grounded in conventional ASR systems.

is the prior probability of the word sequence

W and this probability is provided by the language model which is trained on a large written corpus of the target language. is the likelihood of the observed feature sequence O given the word sequence W

, and this probability is associated with the acoustic model. The acoustic model captures the information about the acoustic component of speech signals, aiming to classify different acoustic units accurately. Traditionally, each phone in the phonetic alphabet is modeled using multiple three-state HMM models for different preceding and following phonetic context (triphone) 

(Lee, 1990). The emission probability of these HMM states are shared (tied) among different models to reduce the number of model parameters (Hwang and Huang, 1993). The output layer of the ANN-based acoustic model is designed accordingly and trained to assign these frame-level tied triphone HMM state (senone) probabilities (Dahl et al., 2012)

. The output layer uses the softmax function to normalize the output into a probability distribution. These values are scaled with the prior probabilities of each class, obtained from the training data, to determine the likelihood values. These likelihood values are later combined with the probabilities assigned by the language model during the decoding stage so as to find the most likely hypothesis.

Speech features, used as inputs to the acoustic model, describe the spectrotemporal dynamics of the speech signal and discriminate among different phones in the target language. Mel-frequency cepstral coefficients(MFCC) (Davis and Mermelstein, 1980)

features are commonly used in conjunction with the GMM-HMM acoustic model. The MFCC features are extracted by (1) performing short-time Fourier transform, (2) applying triangular Mel-scaled filter banks to calculate the power at each Mel frequency in log domain (FBANK) and (3) performing a discrete cosine transform to decorrelate the FBANK features. The third step is often skipped and FBANK features are often used when training ANN-based acoustic models since these models can handle correlation among features. In this work, we incorporate deep SNNs for acoustic modeling instead of the conventional ANNs and compare their ASR performance in different ASR scenarios including phone recognition, low-resourced and standard large vocabulary ASR. The ASR performance obtained using popular speech features have been reported to explore the impact of the feature representation space and its dimensionality for SNN-based acoustic models.

4.3 Speech Recognition with Spiking Neural Network

SNNs are well-suited for representing and processing spatial-temporal signals, they hence possess great potentials for speech recognition tasks. Tavanaei et al. (Tavanaei and Maida, 2017a, b)

proposed SNN-based feature extractors to extract discriminative features from the raw speech signal using unsupervised spiking-timing-dependent plasticity (STDP) rule. While connecting these SNN-based feature extractors with Support Vector Machine (SVM) or Hidden Markov Model (HMM) classifiers, competitive classification accuracies were demonstrated on the isolated spoken digit recognition task. Wu et al.

(Wu et al., 2018a, b)

introduced a SOM-SNN framework for environmental sound and speech recognition. In this framework, the biological-inspired self-organizing map (SOM) is utilized for feature representation, which maps frame-based acoustic features into a spike-based representation that is both sparse and discriminative. The temporal dynamic of the speech signal is further handled by the SNN classifier. Zhang et. al

(Zhang et al., 2019) presented a fully SNN-based speech recognition framework, wherein the spectral information of consecutive frames are encoded with threshold coding and subsequently classified by the SNN that is trained with a novel membrane potential-driven aggregate-labeling learning algorithm.

Recurrent network of spiking neurons (RSNNs) exhibit greater memory capacity than the aforementioned feedforward frameworks. They can capture long temporal information that are useful for speech recognition tasks. In (Zhang et al., 2015), Zhang et al. presented a spiking liquid-state machine (LSM) speech recognition framework which is attractive for low-power very-large-scale-integration (VLSI) implementation. Bellec et al. recently demonstrated state-of-the-art phone recognition accuracy on the TIMIT dataset by adding neuronal adaptation mechanism to the vanilla RSNNs (Bellec et al., 2018). It is the first time that RSNNs approaching the performance of LSTM networks (Greff et al., 2016) on the speech recognition task. These preliminary works on the SNN-based ASR systems are however limited to the phone classification or small vocabulary isolated spoken digit recognition tasks. In this work, we apply deep SNNs to LVCSR tasks and demonstrate competitive accuracies over the comparable ANN-based ASR systems.

5 Methods

5.1 Spiking Neuron Model

As shown in Figure 4, the frame-based features are extracted and input into the SNN-based acoustic models. Given the short temporal duration of segmented frames and the slow variation of speech signals, these features are typically assumed to be stationary over the short time-period of segmented frames. In this work, we use the integrate-and-fire (IF) neuron model with reset by subtraction scheme (Rueckauer et al., 2017), which can effectively process these stationary frame-based features with minimal computational costs. At each time step of a discrete-time simulation, the incoming spikes to neuron at layer are transduced into synaptic current as follows


where indicates the occurrence of an input spike from afferent neuron at time step . In addition, the denotes the synaptic weight that connects presynaptic neuron from layer . Here, can be interpreted as a constant injecting current across the encoding time window of size , and is determined from the bias term of the coupled analog neurons which will be explained in the tandem learning section. As shown in Figure 3, neuron integrates the input current into its membrane potential as per Eq. 5. The is reset and initialized to zero for every new frame-based feature input. Without loss of generality, a unitary membrane resistance is assumed here. An output spike is generated whenever crosses the firing threshold (Eq. 7), which we set to a value of for all the experiments by assuming that all synaptic weights are normalized with respect to the .


According to Eqs. 4 and 5, the free aggregated membrane potential of neuron (no firing) in layer can be expressed as


where is the input spike count from pre-synaptic neuron at layer as per Eq. 9.


The summarizes the aggregate membrane potential contributions of the incoming spikes from pre-synaptic neurons while ignoring their temporal distribution. As will be explained in the tandem learning framework section, this intermediate quantity links the SNN layers to the coupled ANN layers for parameter optimization.

5.2 Neural Coding Scheme

SNNs process information transmitted via spike trains, therefore, special mechanisms are required to encode the continuous-valued feature vectors into spike trains and decode the classification results from the activity of output neurons. To this end, we adopt the spiking neural encoding scheme that proposed in the tandem learning framework (Wu et al., 2019). This encoding scheme first transforms frame-based input feature vector (e.g., MFCC or FBANK features), where

, through a weighted layer of rectified linear unit (ReLU) neurons as follows


where is the strength of the synaptic connection between the input and ReLU neuron . The is the corresponding bias term of the neuron , and

denotes the ReLU activation function. The free aggregate membrane potential

is defined to be equal to the activation value of the ReLU neuron . We distribute this quantity over the encoding time window and represent it with spike trains as per Eqs. 11 and 12.


Altogether, the spike train and spike count that output from the neural encoding layer can be represented as follows


This encoding layer performs weighted transformation inside an end-to-end learning framework. It transforms the original input representation to match the size of the encoding time window and represents the transformed information via spike trains. This encoding scheme is beneficial for rapid inference since the input information can be effectively encoded within a short encoding window. Start from this neural encoding layer, as shown in Figure 4, we input the spike count and to subsequent ANN and SNN layers for tandem learning.

To ensure smooth learning with high precision error gradients derived at the output layer, we use the free aggregate membrane potential of output spiking neurons for neural decoding. Considering that the dimensionality of input feature vectors and output classes are much smaller than that of hidden layers, the computation required will be limited when deploying these two layers onto the edge devices.

5.3 Tandem Learning for Training Deep SNNs

Although IF neurons do not emulate rich temporal dynamics of biological neurons, they are however ideal for working with the neural representation that employed in this work, where spike timings play an insignificant role. It is worth noting that connections are commonly drawn between the activation value of ReLU neurons and the steady-state firing rate of IF neurons (Rueckauer et al., 2017). Here, we present a recently proposed SNN learning rule, under the tandem neural network configuration, that exploits such a connection between the activation value of ANN neurons and the spike count of IF neurons.

By neglecting the temporal dynamic of IF neuron that due to the temporal distribution of incoming spike trains, we may consider the as the main information carrier for SNN layers. The following one-to-one correspondence between the free aggregate membrane potential of spiking neurons and the pseudo ‘spike count’ of artificial neurons can be established.


As shown in Figure 4, during the activation forward propagation, the SNN layers are used to determine the exact spike representation which then propagate the aggregate spike counts and spike trains to the subsequent ANN and SNN layers, respectively. This interlaced layer structure ensures the information that forward propagated to the coupled ANN and SNN layers are synchronized. Taking Eq. 15

as the activation function of ANN layers and using straight-through estimator

(Bengio et al., 2013) to address the discontinuity of the rounding operation, we can use the error gradients derived from ANN layers to approximate those of the coupled SNN layers. It worth noting that the ANN is just an auxiliary structure to facilitates the training of SNN, while only SNN is used during inference.

Notably, competitive classification accuracies are demonstrated with this tandem learning rule for the image classification on the ImageNet dataset

(Wu et al., 2019). By analyzing the relationships between the approximated ‘spike count’ and the actual spike count in a high-dimensional space, Wu et al. have argued that the modified learning dynamic of such a decoupled network can approximate that of an intact ANN. The details of this tandem learning rule are provided in the Algorithm 1.

5.4 SNN-based Acoustic Modeling

To train the deep SNN-based acoustic models, which is the main contribution of this work, several popular speech features have been extracted from the training recordings as described in Section 4.2. Before being fed into the SNNs, these input speech features are contextualized by splicing multiple frames so as to exploit more temporal context information. Before training the SNN-based acoustic model, alignments of the speech features with the target senone labels are obtained using a conventional GMM-HMM-based ASR system similar to that described in (Dahl et al., 2012)

. These frame-level alignments enable the training of the deep SNN acoustic model with the tandem learning approach. During the training, the deep SNN learns to map input speech features to posterior probabilities of senones (cf. Section 

4.2) by passing the input speech frames through multiple layers of spiking neurons.

During the inference phase, the acoustic scores provided by the trained SNN model are combined with the information stored in the language model and pronunciation lexicon. It is a common practice to use the weighted finite state transducers (WFST) (Mohri et al., 2002) as a unified representation of different ASR resources for creating the search graph containing possible hypotheses. The main motivation for using the WFST-based decoding is: (1) the straightforward composition of different ASR resources for constructing a mapping from HMM states to word sequences and (2) the existence of efficient search algorithms operating on WFST that speed up the decoding process. As a result of the search process, the most likely hypotheses are found and stored in the form of a lattice. The ASR output is chosen based on the weighted sum of the acoustic and language model scores belonging to hypotheses in the lattice. For further details of the WFST-based decoding approach used in this work, we refer the reader to (Povey et al., 2012). In the following sections, we describe the ASR experiments conducted to evaluate the recognition performance of the proposed SNN-based acoustic modeling in several recognition scenarios.

5.5 Training and Evaluation

5.5.1 Datasets

The performance of the proposed SNN-based acoustic models is investigated in three different ASR tasks: (1) phone recognition using the TIMIT corpus (Garofolo et al., 1993), (2) low-resourced ASR task using the FAME code-switching Frisian-Dutch corpus (Yılmaz et al., 2016) and (3) standard large-vocabulary continuous ASR task using the Librispeech corpus (Panayotov et al., 2015). All speech data used in the experiments has a sampling frequency of 16 kHz.

The train, development and test sets of the standard TIMIT corpus contain 3,696, 400 and 192 utterances from 462, 50 and 24 speakers, respectively. Each utterance is phonetically transcribed using a phonetic alphabet consisting of 48 phones in total. The training data of the FAME corpus comprises of 8.5 hours and 3 hours of broadcast speech from Frisian and Dutch speakers, respectively. The training utterances are spoken by 382 speakers in total. This bilingual dataset contains Frisian-only and Dutch-only utterances as well as mixed utterances with inter-sentential, intra-sentential and intra-word code-switching (Myers-Scotton, 1989). The development and test sets consist of 1 hour of speech from Frisian speakers and 20 minutes of speech from Dutch speakers each. The total number of speakers is 61 in the development set and 54 in the test set.

The Librispeech corpus contains 1,000 hours of reading speech in total collected from audiobooks. This publicly available has been considered as a popular benchmark for ASR algorithms with multiple training and testing settings. In the ASR experiments, we train acoustic models using the 100 (train_clean_100) and 360 (train_clean_360) hours of speech and apply these models to the clean development (dev_clean) and test (test_clean) sets. Further details about this corpus can be found in (Panayotov et al., 2015).

5.5.2 Implementation Details

All ASR experiments are performed using the PyTorch-Kaldi ASR toolkit 

(Ravanelli et al., 2019). This recently introduced toolkit inherits the flexibility of PyTorch toolkit (Paszke et al., 2017) for ANN-based acoustic model development and the efficiency of Kaldi ASR toolkit (Povey et al., 2011). We implement the SNN tandem learning rule in PyTorch and integrate it into the PyTorch-Kaldi toolkit for training the proposed SNN-based acoustic models (cf. Figure 4

). The PyTorch implementation of the described SNN acoustic models will be made available online soon. For the baseline ANN models, the standard multi-layer perceptron recipes are used. The Kaldi toolkit is used for obtaining the initial alignments, feature extraction, graph creation, and decoding.

For all recognition scenarios, ANNs and SNNs are constructed with 5 hidden layers and 2048 hidden units each using the ReLU activation function. Each fully-connected layer is followed by a batch normalization layer and a dropout layer with a drop probability of 10% to prevent overfitting. We train these models using various popular speech features including the 13-dimensional Mel-frequency cepstral coefficient (MFCC) feature, 23-dimensional Mel-filterbank (FBANK) feature, and higher resolution 40-dimensional MFCC and FBANK features. We further extract feature space maximum likelihood linear regression (FMLLR) 

(Gales, 1998)

features to explore the impact of speaker-dependent features. All features include the deltas and delta-deltas; mean and variance normalization are applied before the splicing. The time context size is set to 11 frames by concatenating 5 frames preceding and following. All features are encoded within a short time window of 10-time steps for SNN simulations.

The neural network training is performed by mini-batch Stochastic Gradient Descent (SGD) with an initial learning rate of 0.08 and a minibatch size of 128. The learning rate is halved if the improvement is less than a preset threshold of 0.001. The final acoustic models of the TIMIT and FAME corpora are obtained after 24 training epochs, while the models of the Librispeech corpus are trained for 12 epochs.

For the TIMIT and Librispeech ASR tasks, we follow the same language model (LM) and pronunciation lexicon preparation pipeline as provided in the corresponding Kaldi recipes222{timit,librispeech}. The smallest 3-gram LM (tgsmall) of the Librispeech corpus is used to create the graph for the decoding stage. The details of the LM and lexicon used in the FAME recognition task are given in (Yılmaz et al., 2018).

5.5.3 Evaluation Metrics

ASR Performance

The phone recognition on the TIMIT corpus is reported in terms of the phone error rate (PER). The word recognition accuracies on the FAME and Librispeech corpora are reported in terms of word error rate (WER). Both metrics are calculated as the ratio of all recognition errors (insertion, deletion, and substitution) and the total number of phones or words in the reference transcriptions.

Energy Efficiency: Counting Synaptic Operations

To compare the energy efficiency of ANN and its equivalent SNN implementation, we follow the convention from NC community and compute the total synaptic operations that required to perform a certain task (Merolla et al., 2014; Rueckauer et al., 2017; Sengupta et al., 2019). For ANN, the total synaptic operations (Multiply-and-Accumulate (MAC)) per classification is defined as follows


where denotes the number of fan-in connections to each neuron in layer , and refers to the number of neurons in layer . In addition, denotes the total number of network layers. Hence, given a particular network configuration, the total synaptic operations required per classification is a constant number that jointly determined by and .

While for SNN, as per Eq. 17, the total synaptic operations (Accumulate (AC)) required per classification are correlated with the spiking neurons’ firing rate, the number of fan-out connections to neurons in the subsequent layer as well as the simulation time window .


where indicates whether a spike is generated by neuron of layer at time instant .

6 Results

6.1 Phone Recognition on TIMIT Corpus

We report the PER on the development and test sets of TIMIT corpus in Table 1, with numbers in bold being the best performance given by the speaker-independent features. ASR performances of other state-of-the-art systems using various ANN and SNN architectures are given in the upper panel for reference purposes. As the results shown in Table 1, the proposed SNN-based acoustic models are applicable to different speech features and provide comparable or slightly worse ASR performance than the ANNs with the same network structure. In particular, the ANN system trained with the standard 13-dimensional FBANK feature achieves the best PER of 16.9% (18.5%) on the development (test) set. The equivalent SNN system using the same feature achieves slightly worse PER of 17.3% (18.7%) on the development (test) set. Although the state-of-the-art ASR systems (Ravanelli et al., 2018) give approximately 1% lower PER than the proposed SNN-based phone recognition system, it is largely credit to the longer time context explored by the recurrent Li-GRU model.

It worth mentioning that phone recognition is still a challenging task for spiking neural networks. To the best of our knowledge, only one recent work with recurrent spiking neural networks (Bellec et al., 2019) demonstrates some promising test results on this corpus with a PER of 26.4%. In contrast, our system has achieved significantly lower PER compared to this preliminary study of SNN-based acoustic modeling. However, these results are not directly comparable since the proposed system incorporates both an acoustic and a language model during decoding unlike the system described in (Bellec et al., 2019).

The experimental results on the TIMIT phone recognition task can be considered as an initial indicator of the compelling prospects of the SNN-based acoustic modeling. Given that the phone recognition task on TIMIT corpus is simplistic compared to the modern LVCSR tasks, we further compare the ANN and SNN performance on newer corpora designed for LVCSR experiments.

6.2 Low-resourced ASR on FAME Corpus

In this section, we apply the SNN-based ASR systems to the low-resourced ASR scenario. As summarized in Table 2, the word recognition results on the FAME corpus are reported separately for monolingual Frisian (fy), monolingual Dutch (nl) and code-switched (cs) utterances. The overall performance (all) is also included in the rightmost column. Given that 8.5 hours Frisian and 3 hours of Dutch speech is used during the training phase, we can compare the ASR performance on different subsets, i.e. fy, nl and cs, to identify the variations in the ASR performance for different levels of low-resourcedness. We omit the results on the development set as they follow a similar pattern to the results on the test set.

In this scenario, the SNN acoustic models consistently provide lower WERs than the ANN models for all speech features. Systems with the FBANK features provide lower WERs than those using MFCC features, which is in line with our observations on the TIMIT corpus. The best performance on the test set is obtained using SNN models trained on 40-dimensional FBANK features with an overall WER of 36.9%. In contrast, the ANN model provides a WER of 39.0% for the same setting, which is relatively 5.4% worse than the SNN model. Moreover, the SNN-based acoustic models achieve a relative improvement of 4.7%, 5.2% and 8.2% on the fy, nl and cs subsets of the test set, respectively. These steady improvements in the recognition accuracies highlight the effectiveness of the SNN-based acoustic modeling in scenarios with limited training data compared to the conventional ANN models. The improved ASR performance with SNNs, in the low-resourced setting, may credit to the noisy weight updates derived by the coupled ANN layers of the tandem learning framework (Wu et al., 2019). It has been recognized that introducing noises into the training stage improves the generalization capability of ANN-based ASR systems (Yin et al., 2015). As a result, the noisy training of the tandem learning is expected to improve the recognition performance in low-resourced scenarios. Further investigation on the impact of this noisy training procedure remains as future work.

6.3 LVCSR experiments on Librispeech Corpus

In the final set of ASR experiments, we train acoustic models using the official 100-hour and 360-hour training subsets of the Librispeech corpus to compare the recognition performance of ANN and SNN models in a standard LVCSR scenario. As the results given in the middle panel of Table 3, for 100 hours of training data, the ANN systems perform marginally better than the corresponding SNN systems across all different speech features. The absolute WER differences range from 0.1% to 0.6%. These marginal performance degradations of the SNN models is likely due to the reduced representation power of using discrete spike counts. Nevertheless, these results are promising even when comparing to the state-of-the-art ASR systems using more complex ANN architectures as provided in the upper panel of Table 3.

It worth noting that both ANN and SNN systems can take benefit of an increased amount of training data. When increasing the training data from 100 hours to 360 hours, the WERs of the best SNN models reduced from 10.0% (10.3%) to 9.2% (9.4%) for the development (test) sets, respectively. To the best of our knowledge, it is the first time that SNN-based acoustic models have achieved comparable results over the ANN models for LVCSR tasks. These results suggest that SNNs are potentially good candidates for acoustic modeling.

6.4 Energy Efficiency of SNN-based ASR Systems

In addition to the promising modeling capability, the SNN-based ASR systems can achieve unprecedented performance gain when implemented on the low-power neuromorphic chips. In this section, we shed light on this prospect by comparing the energy efficiency of ANN- and SNN-based acoustic models. Given that data movements are the most energy-consuming operations for data-driven AI applications, we calculate the average synaptic operations on 5 randomly chosen utterances from the TIMIT corpus and report the ratio of average synaptic operations required per feature classification (SynOps(SNN) / SynOps(ANN)). To investigate the effect of different feature representations, we repeat our analysis on the 40-dimensional MFCC, FBANK and FMLLR features as summarized in Table 4 and Figure 5.

Taking advantage of the short encoding time window (), the sparse neuronal activities are observed for all network layers as shown in Figure 5. Among the three features explored in this experiment, it is interesting to note the FMLLR feature achieves the lowest average spike rate. It is likely due to the more discriminative nature of the speaker-dependent feature, while it worth to note that the FMLLR feature is not always available in all ASR scenarios. As provided in Table 4, the SNN implementations taking MFCC, FBANK and FMLLR input features require 1.72, 1.10 and 0.68 times synaptic operations to their ANN counterparts, respectively. Although the average number of synaptic operations required for SNNs that using MFCC and FBANK features are slightly higher than the ANNs, the AC operations performed on SNNs are much cheaper than the MAC operations required for ANNs. One recent study on the Global Foundry 28 nm process has revealed that MAC operations are 14 times more costly than AC operations and requires 21 times more chip area (Rueckauer et al., 2017). Therefore, when deploying SNNs onto the emerging neuromorphic chips for inference (Merolla et al., 2014; Davies et al., 2018), we expect to receive at least an order of magnitude energy and chip area savings. While the actual energy savings for SNN-based acoustic models are dependent on the chip architectures and materials used, which is outside the scope of this work.

7 Discussion

The remarkable progress in the automatic speech recognition systems has revolutionized the human-computer interface. The rapid growing demands of ASR services have raised concerns on computational efficiency, real-time performance, and data security, etc. It, therefore, motivates novel solutions to address all those concerns. As inspired by the event-driven computation that observed in the biological neural systems, we explore using brain-inspired spiking neural networks for large vocabulary ASR tasks. For this purpose, we proposed a novel SNN-based ASR framework, wherein the SNN is used for acoustic modeling and map the frame-level features into a set of acoustic units. These frame-level outputs will further integrate the word-level information from the corresponding language model to find the most likely word sequence corresponding to the input speech signal.

7.1 Superior Speech Recognition Performance with SNNs

The phone and word recognition experiments on the well-known TIMIT and Librispeech benchmarks have demonstrated the promising modeling capacity of SNN acoustic models and their applicability to different input features. These preliminary results have shown that the recognition performance of SNNs is either comparable or slightly worse than the ANNs with the same network architecture on the TIMIT and Librispeech benchmarks. A possible reason for this performance degradation is the reduced representation power of the discrete neural representation (i.e., spike counts) as compared to the continuous floating-point representation of the ANNs (Wu et al., 2019). This performance gap could potentially be closed by extending the encoding window of SNNs. Moreover, the recognition performance of ANN and SNN models in a low-resourced scenario is also investigated. In this scenario, the SNN acoustic models outperform the conventional ANNs that could be attributed to the noisy training of the tandem learning framework, wherein error gradients of the SNN layers are approximated from the coupled ANN layers.

The neural encoding scheme adopted in this work allows input features to be encoded inside a short encoding time window for rapid processing by SNNs. It is attractive for the time-synchronous ASR tasks that require real-time performance. The preliminary study of the energy efficiency on the TIMIT corpus reveals at least an order of magnitude energy and chip area savings, as compared to the equivalent ANNs, can be achieved when deploying the offline trained SNNs onto neuromorphic chips. The recent study of a keyword spotting task on the Loihi neuromorphic research chip (Blouw et al., 2019) has also demonstrated the compelling energy savings, real-time performance and good scalability of emerging NC architectures over conventional low-power AI chips designed for ANNs.

7.2 Development of SNN-based ASR Systems

The active development of open-source software toolkits plays a significant role in the rapid progress of ASR research, instances include the Kaldi

(Povey et al., 2011) and ESPnet (Watanabe et al., 2018). In this work, we demonstrate that state-of-the-art SNN acoustic models can be easily developed in PyTorch and integrated into the PyTorch-Kaldi Speech Recognition Toolkit (Ravanelli et al., 2019). This software toolkit integrates the efficiency of Kaldi and the flexibility of PyTorch, therefore, it can support the rapid development of SNN-based ASR systems.

7.3 Future Directions

The recurrent neural networks have shown great modeling capability for temporal signals by exploring long temporal context information in the input signals

(Graves and Jaitly, 2014b). As future work, we will explore the recurrent networks of spiking neurons for large-vocabulary ASR tasks to further improve the recognition performance.

The substantial research efforts are devoted to reducing the computational cost and memory footprint of ANNs during inference, instances include network compression (Han et al., 2015), network quantization (Courbariaux et al., 2016; Zhou et al., 2016) and knowledge distillation (Hinton et al., 2015). While the computational paradigm underlying the efficient biological neural networks is fundamentally different from ANNs and hence fosters enormous potentials for neuromorphic computing architectures. Furthermore, grounded on the same connectionism principle, the information of both ANN and SNN are encoded in the network connectivity and connection strength. Therefore, SNN can also take benefits from these early research works on the network compression and quantization of ANNs to further reduce its memory footprint and computation cost (Deng et al., 2019).

The event-driven silicon cochlea audio sensors (Liu et al., 2014) are designed to mimic the functional mechanism of human cochlea and transform input audio signals into spiking events. Given temporally sparse information is transmitted in the surrounding environment, these sensors have shown greater coding efficiency than conventional microphone sensors (Liu et al., 2019). There are some interesting preliminary ASR studies explore the input spiking events captured by these silicon cochlea sensors (Anumula et al., 2018; Acharya et al., 2018). However, the scale of the ASR tasks explored in these studies is relatively small comparing to modern ASR benchmarks due to the limited availability of event-based ASR corpora. Pan et al. (Pan et al., 2019) recently proposed an efficient and perceptually motivated auditory neural encoding scheme to encode the large-scale ASR corpora collected by microphone sensors into spiking events. With this encoding scheme, approximately 50% spiking events can be reduced with negligible interference to the perceptual quality of inputs audio signals. Taking benefits from these earlier research on the neuromorphic auditory front-end, we are expecting to further improve the energy efficiency of SNN-based ASR systems.

The promising initial results demonstrated by the SNN-based large vocabulary ASR systems in this work is the first step towards a myriad opportunities for the integration of state-of-the-art ASR engines into mobile and embedded devices with power restrictions. In the long run, the SNN-based ASR systems are expected to take benefits from ever-growing research on novel neuromoprhic auditory front-end, SNN architectures, neuromorphic computing architectures and ultra-low-power non-volatile memory devices to further improve the computing performance.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Author Contributions

JBW and EY designed and conducted all the experiments. All authors contributed to the results interpretation and writing.


This research is supported by Programmatic Grant No. A1687b0033 from the Singapore Government’s Research, Innovation and Enterprise 2020 plan (Advanced Manufacturing and Engineering domain). JBW is also partially supported by the Zhejiang Lab’s International Talent Fund for Young Professionals.


  • Abdel-Hamid et al. (2014) Abdel-Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., and Yu, D. (2014). Convolutional neural networks for speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22, 1533–1545. doi:10.1109/TASLP.2014.2339736
  • Acharya et al. (2018) Acharya, J., Patil, A., Li, X., Chen, Y., Liu, S.-C., and Basu, A. (2018). A comparison of low-complexity real-time feature extraction for neuromorphic speech recognition. Frontiers in neuroscience 12, 160
  • Anumula et al. (2018) Anumula, J., Neil, D., Delbruck, T., and Liu, S. C. (2018). Feature representations for neuromorphic audio spike streams. Frontiers in Neuroscience 12, 23. doi:10.3389/fnins.2018.00023
  • Bellec et al. (2018) Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., and Maass, W. (2018). Long short-term memory and learning-to-learn in networks of spiking neurons. In Advances in Neural Information Processing Systems. 787–797
  • Bellec et al. (2019) Bellec, G., Scherr, F., Subramoney, A., Hajek, E., Salaj, D., Legenstein, R., et al. (2019). A solution to the learning dilemma for recurrent networks of spiking neurons. bioRxiv doi:10.1101/738385
  • Bengio et al. (2013) Bengio, Y., Léonard, N., and Courville, A. (2013). Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432
  • Blouw et al. (2019) Blouw, P., Choo, X., Hunsberger, E., and Eliasmith, C. (2019). Benchmarking keyword spotting efficiency on neuromorphic hardware. In Proceedings of the 7th Annual Neuro-inspired Computational Elements Workshop (ACM), 1
  • Cao et al. (2015) Cao, Y., Chen, Y., and Khosla, D. (2015). Spiking deep convolutional neural networks for energy-efficient object recognition.

    International Journal of Computer Vision

    113, 54–66
  • Chan et al. (2016) Chan, W., Jaitly, N., Le, Q. V., and Vinyals, O. (2016). Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Proc. ICASSP (IEEE), 4960–4964
  • Courbariaux et al. (2016) Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830
  • Dahl et al. (2012) Dahl, G. E., Yu, D., Deng, L., and Acero, A. (2012). Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing 20, 30–42. doi:10.1109/TASL.2011.2134090
  • Davies et al. (2018) Davies, M., Srinivasa, N., Lin, T. H., Chinya, G., Cao, S. H., Y.and Choday, Dimou, G., et al. (2018). Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82–99
  • Davis and Mermelstein (1980) Davis, S. and Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing 28, 357–366. doi:10.1109/TASSP.1980.1163420
  • Deng et al. (2019) Deng, L., Wu, Y., Hu, Y., Liang, L., Li, G., Hu, X., et al. (2019). Comprehensive snn compression using admm optimization and activity regularization. arXiv preprint arXiv:1911.00822
  • Diehl et al. (2015) Diehl, P. U., Neil, D., Binas, J., Cook, M., Liu, S. C., and Pfeiffer, M. (2015). Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCNN). 1–8. doi:10.1109/IJCNN.2015.7280696
  • Furber et al. (2012) Furber, S. B., Lester, D. R., Plana, L. A., Garside, J. D., Painkras, E., Temple, S., et al. (2012). Overview of the SpiNNaker system architecture. IEEE Transactions on Computers 62, 2454–2467
  • Gales (1998) Gales, M. J. F. (1998).

    Maximum likelihood linear transformations for hmm-based speech recognition.

    Computer Speech & Language 12, 75 – 98. doi:
  • Garofolo et al. (1993) Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., Dahlgren, N. L., et al. (1993). TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1). Philadelphia: Linguistic Data Consortium
  • Gerstner and Kistler (2002) Gerstner, W. and Kistler, W. M. (2002). Spiking neuron models: Single neurons, populations, plasticity (Cambridge University Press)
  • Graves and Jaitly (2014a) Graves, A. and Jaitly, N. (2014a). Towards end-to-end speech recognition with recurrent neural networks. In Proceedings of the 31st International Conference on Machine Learning (ICML) (ACM), 1764–1772
  • Graves and Jaitly (2014b) Graves, A. and Jaitly, N. (2014b). Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning. 1764–1772
  • Graves et al. (2013) Graves, A., Mohamed, A., and Hinton, G. (2013). Speech recognition with deep recurrent neural networks. In Proc. ICASSP. 6645–6649. doi:10.1109/ICASSP.2013.6638947
  • Greff et al. (2016) Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., and Schmidhuber, J. (2016). Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems 28, 2222–2232
  • Han et al. (2015) Han, S., Mao, H., and Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149
  • He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proc. IEEE CVPR. 770–778
  • He et al. (2014) He, T., Fan, Y., Qian, Y., Tan, T., and Yu, K. (2014). Reshaping deep neural network for fast decoding by node-pruning. In Proc. ICASSP. 245–249. doi:10.1109/ICASSP.2014.6853595
  • Hinton et al. (2012) Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A.-R., Jaitly, N., et al. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29, 82–97
  • Hinton et al. (2015) Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531
  • Holmberg et al. (2005) Holmberg, M., Gelbart, D., Ramacher, U., and Hemmert, W. (2005). Automatic speech recognition with neural spike trains. In INTERSPEECH. 1253–1256
  • Hu et al. (2018) Hu, Y., Tang, H., Wang, Y., and Pan, G. (2018). Spiking deep residual network. arXiv preprint arXiv:1805.01352
  • Hwang and Huang (1993) Hwang, M.-Y. and Huang, X. (1993). Shared-distribution hidden markov models for speech recognition. IEEE Transactions on Speech and Audio Processing 1, 414–420
  • Jim-Shih Liaw and Berger (1998) Jim-Shih Liaw and Berger, T. W. (1998). Robust speech recognition with dynamic synapses. In IEEE International Joint Conference on Neural Networks Proceedings (IJCNN). vol. 3, 2175–2179
  • Kröger et al. (2009) Kröger, B. J., Kannampuzha, J., and Neuschaefer-Rube, C. (2009). Towards a neurocomputational model of speech production and perception. Speech Communication 51, 793–809
  • Lang et al. (1990) Lang, K. J., Waibel, A. H., and Hinton, G. E. (1990). A time-delay neural network architecture for isolated word recognition. Neural Networks 3, 23 – 43. doi:
  • Laughlin and Sejnowski (2003) Laughlin, S. B. and Sejnowski, T. J. (2003). Communication in neuronal networks. Science 301, 1870–1874
  • Lee (1990) Lee, K. . (1990). Context-independent phonetic hidden markov models for speaker-independent continuous speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 38, 599–609
  • Lei et al. (2013) Lei, X., Senior, A. W., Gruenstein, A., and Sorensen, J. S. (2013). Accurate and compact large vocabulary speech recognition on mobile devices. In Proc. INTERSPEECH. 662–665
  • Lippmann (1989) Lippmann, R. P. (1989). Review of neural networks for speech recognition. Neural Computation 1, 1–38. doi:10.1162/neco.1989.1.1.1
  • Liu et al. (2019) Liu, S.-C., Rueckauer, B., Ceolini, E., Huber, A., and Delbruck, T. (2019). Event-driven sensing for efficient perception: Vision and audition algorithms. IEEE Signal Processing Magazine 36, 29–37
  • Liu et al. (2014) Liu, S. C., van Schaik, A., Minch, B. A., and Delbruck, T. (2014). Asynchronous binaural spatial audition sensor with 2x64x4 channel output. IEEE Transactions on Biomedical Circuits and Systems 8, 453–464
  • Loiselle et al. (2005) Loiselle, S., Rouat, J., Pressnitzer, D., and Thorpe, S. (2005). Exploration of rank order coding with spiking neural networks for speech recognition. In IEEE International Joint Conference on Neural Networks (IJCNN). vol. 4, 2076–2080 vol. 4. doi:10.1109/IJCNN.2005.1556220
  • McGraw et al. (2016) McGraw, I., Prabhavalkar, R., Alvarez, R., Arenas, M. G., Rao, K., Rybach, D., et al. (2016). Personalized speech recognition on mobile devices. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5955–5959. doi:10.1109/ICASSP.2016.7472820
  • Merolla et al. (2014) Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., et al. (2014). A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673
  • Mohri et al. (2002) Mohri, M., Pereira, F., and Riley, M. (2002). Weighted finite-state transducers in speech recognition. Computer Speech & Language 16, 69–88
  • Monroe (2014) Monroe, D. (2014). Neuromorphic computing gets ready for the (really) big time. Communications of the ACM 57, 13–15
  • Myers-Scotton (1989) Myers-Scotton, C. (1989). Codeswitching with English: types of switching, types of communities. World Englishes 8, 333–346
  • Näger et al. (2002) Näger, C., Storck, J., and Deco, G. (2002). Speech recognition with spiking neurons and dynamic synapses: a model motivated by the human auditory pathway. Neurocomputing 44-46, 937–942
  • Neftci et al. (2019) Neftci, E. O., Mostafa, H., and Zenke, F. (2019). Surrogate gradient learning in spiking neural networks. arXiv preprint arXiv:1901.09948
  • Pan et al. (2019) Pan, Z., Chua, Y., Wu, J., Zhang, M., Li, H., and Ambikairajah, E. (2019). An efficient and perceptually motivated auditory neural encoding and decoding algorithm for spiking neural networks. arXiv preprint arXiv:1909.01302
  • Panayotov et al. (2015) Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audio books. In Proc. ICASSP. 5206–5210. doi:10.1109/ICASSP.2015.7178964
  • Paszke et al. (2017) Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., et al. (2017). Automatic differentiation in pytorch. In NIPS-W
  • Pfeiffer and Pfeil (2018) Pfeiffer, M. and Pfeil, T. (2018). Deep learning with spiking neurons: Opportunities & challenges. Frontiers in Neuroscience 12, 774
  • Povey et al. (2011) Povey, A., D.and Ghoshal, Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., et al. (2011). The kaldi speech recognition toolkit. In IEEE ASRU
  • Povey et al. (2018) Povey, D., Cheng, G., Wang, Y., Li, K., Xu, H., Yarmohammadi, M., et al. (2018). Semi-orthogonal low-rank matrix factorization for deep neural networks. In Proc. Interspeech. 3743–3747. doi:10.21437/Interspeech.2018-1417
  • Povey et al. (2012) Povey, D., Hannemann, M., Boulianne, G., Burget, L., Ghoshal, A., Janda, M., et al. (2012). Generating exact lattices in the WFST framework. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4213–4216
  • Ravanelli et al. (2018) Ravanelli, M., Brakel, P., Omologo, M., and Bengio, Y. (2018).

    Light gated recurrent units for speech recognition.

    IEEE Transactions on Emerging Topics in Computational Intelligence 2, 92–102. doi:10.1109/TETCI.2017.2762739
  • Ravanelli et al. (2019) Ravanelli, M., Parcollet, T., and Bengio, Y. (2019). The pytorch-kaldi speech recognition toolkit. In Proc. ICASSP. 6465–6469. doi:10.1109/ICASSP.2019.8683713
  • Rueckauer et al. (2017) Rueckauer, B., Lungu, I. A., Hu, Y., Pfeiffer, M., and Liu, S. C. (2017). Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience 11, 682. doi:10.3389/fnins.2017.00682
  • Sainath et al. (2013) Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B. (2013). Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In Proc. ICASSP. 6655–6659. doi:10.1109/ICASSP.2013.6638949
  • Sainath and Parada (2015) Sainath, T. N. and Parada, C. (2015). Convolutional neural networks for small-footprint keyword spotting. In Proc. INTERSPEECH. 1478–1482
  • Sengupta et al. (2019) Sengupta, A., Ye, Y., Wang, R., Liu, C., and Roy, K. (2019). Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience 13, 95
  • Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, A., I.and Huang, Guez, A., Hubert, T., et al. (2017). Mastering the game of go without human knowledge. Nature 550, 354
  • Tang et al. (2019) Tang, J., Yuan, F., Shen, X., Wang, Z., Rao, M., He, Y., et al. (2019). Bridging biological and artificial neural networks with emerging neuromorphic devices: Fundamentals, progress, and challenges. Advanced Materials , 1902761
  • Tavanaei et al. (2018) Tavanaei, A., Ghodrati, M., Kheradpisheh, S. R., Masquelier, T., and Maida, A. (2018). Deep learning in spiking neural networks. Neural Networks
  • Tavanaei and Maida (2017a) Tavanaei, A. and Maida, A. (2017a). Bio-inspired multi-layer spiking neural network extracts discriminative features from speech signals. In International Conference on Neural Information Processing (Springer), 899–908
  • Tavanaei and Maida (2017b) Tavanaei, A. and Maida, A. (2017b). A spiking network that learns to extract spike signatures from speech signals. Neurocomputing 240, 191–199
  • Watanabe et al. (2018) Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., et al. (2018). Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.00015
  • Watanabe et al. (2017) Watanabe, S., Hori, T., Kim, S., Hershey, J. R., and Hayashi, T. (2017). Hybrid CTC/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing 11, 1240–1253
  • Wu et al. (2018a) Wu, J., Chua, Y., and Li, H. (2018a). A biologically plausible speech recognition framework based on spiking neural networks. In 2018 International Joint Conference on Neural Networks (IJCNN). 1–8. doi:10.1109/IJCNN.2018.8489535
  • Wu et al. (2019) Wu, J., Chua, Y., Zhang, M., Li, G., Li, H., and Tan, K. C. (2019). A Tandem Learning Rule for Efficient and Rapid Inference on Deep Spiking Neural Networks. arXiv e-prints , arXiv:1907.01167
  • Wu et al. (2018b) Wu, J., Chua, Y., Zhang, M., Li, H., and Tan, K. C. (2018b). A spiking neural network framework for robust sound classification. Frontiers in neuroscience 12
  • Wu et al. (2018) Wu, M., Panchapagesan, S., Sun, M., Gu, J., Thomas, R., Prasad Vitaladevuni, S. N., et al. (2018). Monophone-based background modeling for two-stage on-device wake word detection. In Proc. ICASSP. 5494–5498. doi:10.1109/ICASSP.2018.8462227
  • Wu et al. (2018) Wu, Y., Deng, L., Li, G., Zhu, J., and Shi, L. (2018). Direct training for spiking neural networks: Faster, larger, better. arXiv preprint arXiv:1809.05793
  • Xiong et al. (2017) Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., et al. (2017). Toward human parity in conversational speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25, 2410–2423. doi:10.1109/TASLP.2017.2756440
  • Xue et al. (2013) Xue, J., Li, J., and Gong, Y. (2013).

    Restructuring of deep neural network acoustic models with singular value decomposition.

    In Proc. Interspeech. 2365–2369
  • Yılmaz et al. (2016) Yılmaz, E., Andringa, M., Kingma, S., Van der Kuip, F., Van de Velde, H., Kampstra, F., et al. (2016). A longitudinal bilingual Frisian-Dutch radio broadcast database designed for code-switching research. In Proc. LREC. 4666–4669
  • Yılmaz et al. (2016) Yılmaz, E., van den Heuvel, H., and van Leeuwen, D. (2016). Code-switching detection using multilingual DNNs. In 2016 IEEE Spoken Language Technology Workshop (SLT). 610–616. doi:10.1109/SLT.2016.7846326
  • Yılmaz et al. (2018) Yılmaz, E., Van den Heuvel, H., and Van Leeuwen, D. A. (2018). Acoustic and textual data augmentation for improved ASR of code-switching speech. In Proc. INTERSPEECH. 1933–1937
  • Yin et al. (2015) Yin, S., Liu, C., Zhang, Z., Lin, Y., Wang, D., Tejedor, J., et al. (2015). Noisy training for deep neural networks in speech recognition. EURASIP Journal on Audio, Speech, and Music Processing 2015, 2
  • Yu and Deng (2015) Yu, D. and Deng, L. (2015). Automatic Speech Recognition: A Deep Learning Approach. Signals and Communication Technology (Springer)
  • Zehetner et al. (2014) Zehetner, A., Hagmuller, M., and Pernkopf, F. (2014). Wake-up-word spotting for mobile systems. In Proc. EUSIPCO. 1472–1476
  • Zhang et al. (2019) Zhang, M., Wu, J., Chua, Y., Luo, X., Pan, Z., Liu, D., et al. (2019). Mpd-al: an efficient membrane potential driven aggregate-label learning algorithm for spiking neurons. In

    Proceedings of the AAAI Conference on Artificial Intelligence

    . vol. 33, 1327–1334
  • Zhang et al. (2015) Zhang, Y., Li, P., Jin, Y., and Choe, Y. (2015). A digital liquid state machine with biologically inspired learning and its application to speech recognition. IEEE transactions on neural networks and learning systems 26, 2635–2649
  • Zhou et al. (2016) Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. (2016). Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160


Input: Input frame-based feature vectors , target label , network parameters , neural encoding window size
Output: Updated network parameters
Forward Pass:
= Neural_Encoding()
for layer = 1 to N-1 do
        // State Update of the ANN Layer
       = ANN.layer[].forward(, )
       for t = 1 to  do
             // State Update of the SNN Layer
             = SNN.layer[l].forward(, )
       // Update the Spike Count
/* Neural Decoding with the Aggregate Membrane Potential */
= ANN.layer[N].forward(, )
Loss: = LossFunction()
Backward Pass:
= LossGradient()
for layer = N-1 to 1 do
        // Gradient Update through the ANN Layer
       , = ANN.layer[].backward( , , )
Update parameters of the ANN layer based on the calculated gradients.
Copy the updated parameters to the corresponding SNN layer.
For inference, state updates are performed on the SNN layers entirely.
Algorithm 1 Pseudo Codes For The Tandem Learning Rule


FeaturesAM Li-GRU(Ravanelli et al., 2018) RSNN(Bellec et al., 2019)
MFCC (13-dim.) 16.7 26.4
FBANK (13-dim.) 15.8 -
FMLLR   14.9* -
Dev Test
MFCC (13-dim.) 17.1 17.8 18.5 19.1
FBANK (13-dim.) 16.9 17.3 18.5 18.7
MFCC (40-dim.) 17.3 18.2 18.7 19.8
FBANK (40-dim.) 16.9 17.8 17.9 19.1
FMLLR 15.8 16.5 17.2 17.4
Table 1: PER (%) on the TIMIT development and test sets. The upper panel reports the results of various ANN and SNN architectures from the literatures, and the lower panel presents the results achieved by the ANN and SNN models in this work (AM: acoustic model, *: the best result to date).
fy nl cs all
# of Frisian words 10753 0 1798 12551
# of Dutch words 0 3475 306 3781
Speech features AM
FBANK (40-dim.) Kaldi-ANN(Yılmaz et al., 2016) 32.4 39.7 49.9 36.2
MFCC (40-dim.) TDNN-LSTM(Yılmaz et al., 2018) 31.5 39.5 47.9 35.2
MFCC (13-dim.) ANN 34.6 50.0 49.9 39.9
MFCC (13-dim.) SNN 33.8 45.3 47.9 38.2
FBANK (13-dim.) ANN 34.3 47.5 48.1 39.0
FBANK (13-dim.) SNN 33.1 44.3 46.5 37.3
MFCC (40-dim.) ANN 35.2 48.4 51.7 40.2
MFCC (40-dim.) SNN 33.7 44.2 46.9 37.7
FBANK (40-dim.) ANN 34.4 46.3 49.8 39.0
FBANK (40-dim.) SNN 32.8 43.9 45.7 36.9
FMLLR ANN 31.2 42.1 47.2 35.7
FMLLR SNN 31.5 39.5 46.6 35.2
Table 2: WERs (%) achieved on the monolingual and mixed segments of the FAME test set. The upper panel summarizes the number of words from each language subset. The middle panel provides the results of state-of-the-art ANN achitectures (Yılmaz et al., 2016, 2018) for reference purposes and the lower panel presents the results achieved by the ANN and SNN models in this work (AM: acoustic model).
Train - 100h
Dev Test
Kaldi p-norm ANN 9.2 9.7
PyTorch-Kaldi Li-GRU - 8.6
MFCC 10.3 10.5 10.6 10.9
FBANK 9.6 10.0 10.2 10.6
MFCC (40-dim.) 9.5 10.1 10.0 10.6
FBANK (40-dim.) 9.6 10.2 10.1 10.3
FMLLR 9.2 9.3 9.7 9.9
Train - 360h
Dev Test
MFCC 9.2 9.9 9.6 10.3
FBANK 8.6 9.7 9.1 10.0
MFCC (40-dim.) 8.6 9.2 8.9 9.4
FBANK (40-dim.) 8.5 9.4 8.9 9.7
FMLLR 8.4 9.2 8.8 9.7
Table 3: WER (%) achieved on the Librispeech development and test sets. The upper panel gives the results, with 100-hour of training data, reported at the Github repo of Kaldi and PyTorch-Kaldi. The middle and lower panel present the results achieved by ANN and SNN models in this work using 100-hour and 360-hour of training data, respectively. (AM: acoustic model,: reported at Github repo)
Utterance Index 1 2 3 4 5 Avg. SynOps Ratio
Num. of frames 474 287 274 268 223
MFCC (40-dim.) 1.71 1.73 1.76 1.71 1.68 1.72
FBANK (40-dim.) 1.08 1.08 1.14 1.09 1.10 1.10
FMLLR 0.67 0.68 0.71 0.66 0.67 0.68
Table 4: Comparison of the computational costs between SNN and ANN. The ratio of their required total synaptic operations (SynOps(SNN) / SynOps(ANN)) is reported. It worth mentioning that ANNs use more costly MAC operations than the AC operations used in the SNNs.


Figure 1: Comparison of the synchronous and asynchronous computational paradigms adopted by (A) ANNs and (B) SNNs, respectively (revised from (Pfeiffer and Pfeil, 2018)).

Figure 2: Block diagram of a conventional ASR system. The acoustic and linguistic components are incorporated to jointly determine the most likely hypothesis.

Figure 3: The neuronal dynamic of an integrate-and-fire neuron (red). In this example, three pre-synaptic neurons are sending asynchronous spike trains to this neuron. Output spikes are generated when the membrane potential crosses the firing threshold (top right corner).

Figure 4: System flowchart for SNN training within a tandem neural network, wherein SNN layers are used in the forward pass to determine the spike count and spike train. The ANN layers are used for error back-propagation to approximate the gradient of the coupled SNN layers.

Figure 5: Average spike count per neuron of different SNN layers on the TIMIT corpus. The results of different input features are color-coded. Sparse neuronal activities can be observed in this bar chart.