Log In Sign Up

Espresso: A Fast End-to-end Neural Speech Recognition Toolkit

We present Espresso, an open-source, modular, extensible end-to-end neural automatic speech recognition (ASR) toolkit based on the deep learning library PyTorch and the popular neural machine translation toolkit fairseq. Espresso supports distributed training across GPUs and computing nodes, and features various decoding approaches commonly employed in ASR, including look-ahead word-based language model fusion, for which a fast, parallelized decoder is implemented. Espresso achieves state-of-the-art ASR performance on the WSJ, LibriSpeech, and Switchboard data sets among other end-to-end systems without data augmentation, and is 4--11x faster for decoding than similar systems (e.g. ESPnet).


page 1

page 2

page 3

page 4


KoSpeech: Open-Source Toolkit for End-to-End Korean Speech Recognition

We present KoSpeech, an open-source software, which is modular and exten...

PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR

We present PyChain, a fully parallelized PyTorch implementation of end-t...

RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition

We compare the fast training and decoding speed of RETURNN of attention ...

A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies

In this study, we present recent developments of models trained with the...

BCN2BRNO: ASR System Fusion for Albayzin 2020 Speech to Text Challenge

This paper describes joint effort of BUT and Telefónica Research on deve...

Fast and Accurate OOV Decoder on High-Level Features

This work proposes a novel approach to out-of-vocabulary (OOV) keyword s...

WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit

Recently, we made available WeNet, a production-oriented end-to-end spee...

1 Introduction

Various open-source toolkits for building automatic speech recognition (ASR) systems have been created, with a notable example being Kaldi [42], a weighted finite state transducer based framework with extensive linear algebra support, that enables traditional hybrid ASR systems [56].

With advances in deep learning, recent work in ASR begun paying attention to so-called neural end-to-end systems [16, 9, 8, inter alia], which are characterized by generally smaller code size, and greater portability and maintainability across hardware platforms and software environments. This shift is analogous to the one in the machine translation (MT) community: from feature- and syntax-based statistical machine translation (SMT) systems (e.g. Moses [27], Joshua [30]) to end-to-end neural machine translation (NMT) systems (e.g. OpenNMT [25], OpenSeq2Seq [28], fairseq [36]). As a result, there have been a few efforts in the ASR research community to create open source neural end-to-end frameworks, most notably ESPnet [54] (See also, Table 1). However, ESPnet has some important shortcomings: (i) the code is not very easily extensible and has portability issues due to its mixed dependency on two deep learning frameworks PyTorch [39] and Chainer [51]; (ii) the decoder, which uses a simple but relatively slow beam search algorithm, is not fast enough for quick turnaround of experiments.

max width= Name Language Deep Learning Framework Recipes Other Applications EESEN [34] C++ WSJ, LibriSpeech, SWBD, TED-LIUM, HKUST ESPnet [54] Python Chainer & PyTorch various TTS, ST E2E LF-MMI [18] C++ Kaldi various lingvo [48] Python TensorFlow LibriSpeech NMT OpenSeq2Seq [28] Python TensorFlow LibriSpeech NMT RETURNN [58] Python Theano, TensorFlow WSJ, LibriSpeech, SWBD, CHiME NMT wav2letter++ [44] C++ ArrayFire WSJ, LibriSpeech, TIMIT Espresso Python PyTorch WSJ, LibriSpeech, SWBD NMT

Table 1: Popular end-to-end neural ASR systems and our system.

To address these problems, we present Espresso, a novel neural end-to-end system for ASR.111 Espresso builds upon the popular NMT framework fairseq, and the flexible deep learning framework PyTorch. By extending fairseq, Espresso inherits its excellent extensibility: new modules can easily be plugged into the system by extending standard PyTorch interfaces. Additionally, we gain ability to perform distributed training over large data sets for ASR.

We also present the first fully parallelized decoder for end-to-end ASR, with look-ahead word-based language model fusion [21], tightly integrated with the existing sets of optimized inference algorithms (e.g. beam search) inherited from fairseq and tailored to the scenario of speech recognition. Furthermore, an improved coverage mechanism is proposed to further reduce deletion and insertion errors, and is compared with related techniques such as EOS threshold [19]. Espresso provides recipes for a variety of benchmark ASR data sets, including WSJ [40], LibriSpeech [37], and Switchboard [14], and achieves state-of-the-art results on these data sets.

Espresso, by building upon fairseq

, also has the potential to integrate seamlessly with sequence generation systems from natural language processing (NLP), such as neural machine translation and dialog systems. We envision that

Espresso could become the foundation for unified speech + text processing systems, and pave the way for future end-to-end speech translation (ST) and text-to-speech synthesis (TTS) systems, ultimately facilitating greater synergy between the ASR and NLP research communities.

2 Software Architecture and Design Choices

We implement Espresso with the following design goals in mind:

  • Pure Python / PyTorch that enables modularity and extensibility;

  • Parallelization and distributed training and decoding for quick turnaround of experiments;

  • Compatibility with Kaldi / ESPnet data format to enable reuse of previous / proven data preparation pipelines;

  • Easy interoperability with the existing fairseq codebase to facilitate future joint research areas between speech and NLP.

We elaborate our technical rationale in the following sections.

2.1 Input format and dataset classes

Our speech data follows the format in Kaldi, where utterances are stored in the Kaldi-defined SCP format, consisting of space-delimited lines that follows this template:

<UttID> <FeatureFile>:<Offset>

where <UttID> is the utterance ID, a key that points to any utterance in the dataset, and <FeatureFile> is a string interpreted as an extended filename for reading from a binary file (ARK format) storing the actual acoustic feature data, following the practice222 in Kaldi.

In theory, any kind of acoustic feature vectors (e.g. MFCC) can be stored in the feature file. In

Espresso, we follow ESPnet and employ the commonly used 80-dimensional log Mel feature with the additional pitch features (in total, 83 dimensions for each frame).

In fairseq, there is a concept called “datasets”, which contains a set of training samples and abstracts away details such as shuffling and bucketing. We follow this and create the following dataset classes in Espresso:

  • data.ScpCachedDataset

    : this contains the real-valued acoustic features extracted from the speech utterance. Each training batch drawn from this dataset is a real-valued tensor of shape

    that will be fed to the neural speech encoder (Section 2.3

    ). Since the acoustic features are large and cannot be loaded into memory all at once, we also implement sharded loading, where given the order of the incoming examples in an epoch, a bulk of features is pre-loaded once the previous bulk is consumed for training / decoding. This helps balance the file system’s I/O load and the memory usage.

  • data.TokenTextDataset: this contains the gold speech transcripts as text. Each training batch is a integer-valued tensor of shape , where each integer in this tensor is the index of the character / subword unit in the token dictionary (see below).

  • data.SpeechDataset: this is a container for the two datasets above: each sample drawn from this dataset contains two fields, source and target, that points to the speech utterance and the gold transcripts respectively.

Note that in speech recognition, the token dictionary (set of all vocabulary) is different from the common practice in fairseq due to the additional special token <space>. For this reason, we do not directly use the data.Dictionary class from fairseq, instead, we inherit that class and create our data.TokenDictionary class for this purpose, with the extra functionality of handling <space>.

For speech decoding purposes rather than NMT (default in fairseq), normally the output unit for each decoding step is a subword unit instead of a word, since it is shown that for ASR using whole words as modeling units is only possible when large amounts of training data (at least tens of thousands of hours) is available [49, 2]. A subword unit can either be a character or character sequence like BPE [47] or a SentencePiece333 [29]. Both are supported in Espresso and experimental results will follow.

2.2 Output format

Espresso supports two output format: a raw format and a more detailed aligned results version that helps debugging.

The raw format just consists of space-delimited lines that follows this template:

<UttID> <DecodedSequence>

where <UttID> is the original utterance ID from the SCP dataset, and the <DecodedSequence> is the raw output of the speech recognition system.

The aligned results provide an aligned sequence between the gold reference transcript and the predicted hypothesis. An example is shown below:

  STP:                   D  S   S
  WER: 42.86%

Each such record consists of 5 rows: the first line is the utterance ID; REF and HYP is the reference transcript and the system output hypothsis respectively – these two are aligned using minimal edit distance. The fourth line, STP (step), contains the error the system makes at each decoding step: it could be one of S (substitution), I (insertion) and D (deletion), corresponding to the three types of errors when evaluating the word error rate (WER) commonly used to evaluate speech recognition systems. The last line is WER calculated on this utterance. Such output format facilitates easy human inspection to the different error types made by the system, rendering debugging easier for researchers.

2.3 Encoder-Decoder

Espresso supports common sequence generation models and techniques arisen from the research in the ASR and NLP community. The de facto standard model, the encoder-decoder with attention [3, 32] (also successfully pioneered by [9] in the speech community), is implemented as our models.speech_lstm.SpeechLSTMModel class. Owing to the modularity and extensibility of Espresso, other models, e.g., Transformer [53], can be easily integrated from the underlying fairseq.

CNN-LSTM Encoder

  The default encoder we used is a 4-layer stacked 2-dimensional convolution (with batch normalization between layers), with kernel size

on both the time frame axis and the feature axis [60, 54]. 2-downsampling is employed at layer 1 and 3, resulting in time frames after convolution. The final output channel dimensionality is 128, with the 21 downsampled frequency features, hence a total of dimensional features for each time frame.

Then 3 layers of bidirectional LSTMs [15] are stacked upon the output channels yielded by the stacked convolution layers.

This architecture, with the various dimensionality, number of layers, and other hyperparameters customizable, is implemented in our

models.speech_lstm.SpeechLSTMEncoder class.

LSTM Decoder with Attention  We use a 3-layer LSTM decoder by default, with Bahdanau attention [3] on the encoded hidden states (Luong attention [32] is also implemented). We follow the architecture in the Google Neural Machine Translation (GNMT) system [55]

, where the context vectors generated by the attention mechanism is fed to all 3 layers of the decoding LSTM. Residual connections

[20] are added between the decoder layers. These are implemented in the models.speech_lstm.SpeechLSTMDecoder class.

2.4 Training Strategies

Scheduled Sampling Scheduled sampling [6] is supported by our system, with promising results in end-to-end speech recognition [5]. With scheduled sampling, at each decoder step, the gold label is fed to the next step with probability, whereas the predicted token444

 This is the token with the maximum posterior probability resulting from the previous LSTM decoder step. It may not necessarily be gold.

is fed with probability. In our implementation such mechanism will start at an intermediate epoch in the training process: the first few epochs are always trained with gold labels. The probability can be scheduled in the training process: later epochs might use a smaller probability to encourage the model not to rely on the gold labels.

Label Smoothing Label smoothing [50] has been proposed to improve accuracy by computing the loss (i.e., cross entropy here) not with the “hard” (one-hot) targets from the dataset, but with a weighted mixture of these targets with some distribution [35]. We support three kinds of these distributions in Espresso:

  • Uniform smoothing [50]: The target is a mixture of probability of the one-hot target and the rest of the

    probability mass uniformly distributed across the vocabulary set;

  • Unigram smoothing [41]: Mixed with a unigram language model trained on the gold transcripts;

  • Temporal smoothing [10]: Mixed with a distribution assigning probability mass to neighboring tokens in the transcript. Intuitively, this smoothing scheme helps the model recover from beam search errors: the network is more likely to make mistakes that simply skip a subword unit of the transcript.

Model Selection via Validation At the end of each training epoch, we compute the WER on the validation set using greedy-search decoding without language model fusion (see Section 3). This is different from the approach in previous frameworks such as ESPnet and fairseq

, where they compute the loss function on the validation set (the gold labels are fed in) to perform model selection. We argue that our approach may be more suitable since free decoding on the validation set is a closer scenario to the final metric on test sets. Owing to efficiency concerns, we do not use full-blown language model fused beam search decoding for validation (arguably this is even better).

We employ learning rate scheduling following fairseq: at the end of an epoch, if the metric on the validation set is not better than the previous epoch, the learning rate is reduced by a factor (e.g. ). Empirically we found that the reduction of the learning rate will be less frequent if using WER as the validation metric as compared to the loss value on the validation set. According to [38], a less frequent learning rate reduction generally leads to better performance.

3 Language Model-Fused Decoding

It is shown in recent research that a pure sequence-to-sequence transduction model for ASR without an external language model component (which is used in traditional hybrid ASR systems) is far from satisfactory [23]. This is in contrast with neural machine translation (NMT) models, where normally no external language model is needed. This performance gap is hypothesized to be caused by the fact that the ASR model is only trained on speech-transcript pairs. The gold transcript set is not large enough to produce state-of-the-art neural language models, which are typically trained on a corpus on the scale of 1 billion words.

In Espresso, we employ shallow fusion [17] as a language model integration technique, which is proven to be effective in speech recognition [23, 52]. The LSTM decoder with shallow fusion computes a weighted sum of two posterior distributions over subword units: one from the end-to-end speech recognition model, the other from the external neural language model.

We support 3 types of external neural language models:

  • Subword-unit language model: A vanilla LSTM-based language model trained on subword units. Here subword units can either just be characters (with <space> as an additional special token) or trained subword units (e.g. BPE [47] or SentencePiece [29]);

  • Multi-level language model [22]: This is a combination of character-based and word-based language models. Hypotheses in the beam are first scored with the character-based language model until a word boundary (<space>) is encountered. Known words are then re-scored using the word-based language model, while the character-based language model provides for out-of-vocabulary scores;

  • Look-ahead word-based language model [21]: This model enables outputting characters for each decoding step with a pre-trained word-based language model, by providing look-ahead word probabilities based on the word prefix (sequence of characters) decoded. This is shown in [21] to be superior to the multi-level language model.

3.1 Parallelization with Look-ahead Word-based LMs

The original implementation of the look-ahead word-based language model in ESPnet [54] is not operating on batches, making the decoding speed slow. In Espresso, we devise a fully-parallelized version of the decoding algorithm on GPUs.

In [21], a word-based language model is converted to a character-based one via a technique using prefix trees. The prefix tree automaton is a finite-state automaton (see Fig. 1):

  • is the character set (including <space>);

  • is the word vocabulary set and also the final state set;

  • is the set of all prefixes555 We denote “p is a prefix of q” as : e.g. “stand” “standard”. of the words in and also the state set;

  • is the empty string, which also serves as the initial state;

  • is the state transition function, where given a state and an input character, , i.e. a simple concatenation.

A look-ahead word-based LM computes the probability of the next character based on a given word history and a word prefix (i.e., a state in the prefix tree automaton):


where is the probability of the word w predicted by the word-based LSTM language model. In Eq. (1), the numerator is the sum of the probability of all words prefixed by , i.e. all possible words that could be generated from if the state is moved from to ; the denominator is the sum of the probability of all possible words at the current state (see Fig. 1).

[21] proposed an efficient way to compute the sum in Eq. (1). We denote p precedes q lexicographically as , and define the upper bound (the lexicographically greatest element prefixed by p) and lower bound (the greatest element lexicographically less than any word prefixed by p) as:


Given that the vocabulary set is sorted lexicographically, we can efficiently compute the sum of the probability of all words preceding or equal to a given word, using efficient routines like cumsum:


Using these definitions, Eq. (1) can be rewritten as


This method is illustrated in Fig. 1, showing that the probability of a character given a prefix can be efficiently computed via a cumsum operation and simple arithmetics.

In our parallelized implementation, each prefix (corresponding to a state in the automaton) is indexed as a unique integer. Hence the batch of decoding states is compactly stored as a vector of integers, each corresponding to a state on the prefix tree automaton. The automaton itself is stored as a matrix with shape , where the row contains the index of all descendants of p, logically forming an adjacency list. The index of the and for each state p is also precomputed and cached. In sum, the entire prefix tree automaton is vectorized.

To compute for all over batches and beam hypotheses, the following steps are executed:

  1. [label=()]

  2. Update for all from a specific decoding step of the word-based language model for those hypotheses that encounter the end-of-word (<space>) symbol in the batch;

  3. Update the function using for all ;

  4. Get all possible successive states;

  5. Get all upper and lower bounds for all successive states;

  6. Compute the probability for each according to Eq. (4).

Note that the first step follows a batched forward computation of a neural language model; the third and the fourth steps can be computed via tensorized advanced indexing; and the second and the last steps can be executed using vectorized arithmetics. Hence we obtain a fully parallelized algorithm that runs on GPUs.

Figure 1: The efficient look-ahead LM algorithm. The dark gray and light gray subtrees correspond to the probability mass spanned by prefix and (numerator and denominator in Eq. (1)) respectively. These can be efficiently computed via Eq. (4) using cumsum.

As mentioned in the first step, at a specific decoding step, some elements in the batch may encounter the end-of-word symbol, whereas others may not: running this conditional operation requires special treatment. We devise an algorithm that shares the spirit with [13], an parallelized algorithm for stack LSTM parsing: first we record the elements in the batch that has not reached the end of a word with a mask, then progress the state for one step for all of these elements, finally, for those masked states, their original states are restored, and only of those not masked are updated.

3.2 Improved Coverage

With language model fusion, the decoder tends to make more deletion errors when the language model weight becomes larger [4]. A coverage term (a scalar value assigned to each hypothesis in the beam) is first proposed in [10] to promote longer transcripts and also to prevent attentions from looping over utterance (repeating certain -grams when decoding) when using shallow fusion:


where is the attention weight for the decoder step and encoder frame , is a tunable hyper-parameter, and is the indicator function. This is the total number of encoder frames that has been sufficiently “attended” to. However, based on our experiments, applying the coverage term Eq. (5) is not sufficient to prevent words from being repeated. Instead we propose a modified version of the coverage term which penalizes looping attentions more aggressively:

coverage (6)

where are tunable hyper-parameters. While the first term in Eq. (6) is exactly the same as Eq. (5), the second term penalizes the hypothesis score when the accumulated attention on encoder frame exceeds . Specifically, if the accumulated attention weight on frame exceeds but not , only the first term is activated, increasing the coverage score to encourage more attention on frame ; if the accumulated attention weight further exceeds , the second term (with the minus sign) is also activated and its magnitude is the amount of the exceeding value plus a constant , discouraging further attention accumulated on the same frame. Therefore, the new coverage mechanism enforces a soft constraint on the accumulated attention weight on each frame to be between and , leading to both less deletion errors and less repeating -grams (shown in the WSJ part of Section 4). During beam search decoding this new coverage term as a whole is added to the hypothesis score with a weight (e.g. 0.01). In our experiments, .

3.3 EOS Threshold

We implement the end-of-sentence threshold technique proposed in [19] to bias the decoder away from short transcriptions when decoding with a fused language model. End-of-sentence (<eos>) tokens can only be emitted if its probability is greater than a specific factor of the top output token candidate during beam search:


where is the vocabulary set. In our experiments, is set to 1.5.

4 Recipes and Results

Espresso provides running recipes for a variety of well-known data sets. We elaborate the details of our recipes on Wall Street Journal [40] (WSJ), an 80-hour English newspaper speech corpus, LibriSpeech [37], a corpus of approximately 1,000 hours of read English speech, and Switchboard [14] (SWBD), a 300-hour English telephone speech corpus.

Besides the transcripts, all of these data sets have their own extra text corpus for training language models. Input and output word embeddings are tied [45] to reduce model size. All the models are optimized using Adam [24] with an initial learning rate , and then halved if the metric on the validation set at the end of an epoch does not improve over the previous epoch. The training process stops if the learning rate is less than . Curriculum learning [7], which is quite helpful for preventing training divergence and improving performance, is employed for the first 1 (LibriSpeech) or 2 (WSJ / SWBD) epochs. All the models are trained / evaluated using NVIDIA GeForce GTX 1080 Ti GPUs. If not otherwise specified, all the models throughout this paper are trained with 2 GPUs using fairseq built-in distributed data parallellism. Note that no data augmentation techniques such as speed-perturbation [26] or the more recent SpecAugment [38] is applied.

The hyper-parameters for the recipes are listed in Table 2.

max width= Hyper-parameter WSJ LibriSpeech SWBD LM ASR LM ASR LM ASR Vocab. size 65k 52 5k 5k 1k 1k Encoder # layers - 3 - 4 - 4 Decoder # layers 3 3 4 3 3 3 Emb. dim. 1,200 48 800 1,024 1,800 640 Hidden dim. 1,200 320 800 1,024 1,800 640 # Params. 113M 18M 25M 174M 80M 70M Dropout rate 0.35 0.4 0.0 0.3 0.3 0.5 Avg. batch size 435 48 1,704 34 1,783 69 Beam size 50 40 35 LM fusion weight 0.9 0.42 0.25

Table 2: Hyper-parameters for the three recipes.

WSJ We adopt the look-ahead word-based language model [21] as the external language model. We report the perplexities on the validation / evaluation set: The external word-based language model achieves 72.0 perplexity on dev93 and 59.0 on eval92.

For the encoder-decoder model, the vocabulary size of subword units (characters) for WSJ is 52, the same as in ESPnet.666 It includes 45 characters constituting the training transcripts, plus 3 atomic symbols: <*IN*>, <*MR.*> and <NOISE>, plus 4 special symbols:


, <eos>, <unk> and <space>. Temporal label smoothing with and scheduled sampling with starting at epoch 6 are adopted.

Baseline end-to-end systems are compared: Hadian et al. [18], an end-to-end777 This is a hybrid system. “End-to-end” here is in the sense that it does not need HMM-GMM training or tree-building steps. model with the lattice-free MMI objective [43]; Baskar et al. [5], an encoder-decoder model with discriminative training in ESPnet; Zeghidour et al. [57], a pure convolutional network with ASG loss [11]; Likhomanenko et al. [31]

, a lexicon-free decoding method with the acoustic model proposed in

[57]; and the last one, Deep Speech 2 [1].

We show the beam search decoding results of various configurations of Espresso in Table 3 with beam size 50. The breakdown of the three kinds of errors is shown in Table 4. The first row gives WERs where no language model fusion is applied. The second row is after integrating the look-ahead word-based language model, with its optimal LM fusion weight 0.5. Although it has already significantly reduced the overall WER, deletion errors increase. Further applying the improved coverage yields better performance by suppressing deletion errors. If we only use the first term in Eq. (6) which is equivalent to the coverage term in [10], the insertions errors will increase from 0.8 to 1.3 on dev93, and from 0.6 to 0.9 on eval92. A manual inspection of the decoded results reveals that these additional insertions are mostly repeated words. This observation validates the effectiveness of our improved coverage mechanism. Alternatively, if we apply the EOS threshold, we achieve state-of-the-art performance on WSJ among end-to-end models.

max width= dev93 eval92 Hadian et al.888 The result is with “full bichar” (using all possible 2-gram characters as context-dependent modeling units), data speed-perturbation, and a 3-gram word language model. [18] - 4.1 Baskar et al. (ESPnet) [5] - 3.8 Likhomanenko et al. [31] 6.4 3.6 Zeghidour et al. [57] 6.8 3.5 Amodei et al.999 It uses 12,000 hours transcribed data for acoustic model training and large crawled text for language model training, making it not comparable. (Deep Speech 2) [1] 4.4 3.1 Espresso LSTM 14.8 12.1  +Look-ahead Word LM 7.4 5.1   +Improved Coverage 5.9 3.5   +EOS Threshold 5.9 3.4

Table 3: WERs (%) on the WSJ dev93 and eval92 set.

max width= dev93 eval92 Sub Ins Del Sub Ins Del Espresso LSTM 12.0 1.4 1.4 9.7 1.5 1.0  +Look-ahead Word LM 4.6 0.8 2.0 3.1 0.7 1.3   +Improved Coverage 4.3 0.8 0.8 2.7 0.6 0.3   +EOS Threshold 4.1 0.9 0.8 2.6 0.5 0.3

Table 4: Breakdown of the WERs (%) on WSJ.

LibriSpeech SentencePiece is used as the subword units in our LibriSpeech setup for both external language modeling and encoder-decoder model. We combine dev-clean and dev-other sets together as a single validation set for both language model and encoder-decoder model training.

Again, we report the perplexities on the validation / evaluation sets. Espresso obtains 35.8 and 37.5 on the dev-clean and dev-other sets, and 37.7 and 37.8 on test-clean and test-other.

Uniform smoothing with is applied throughout the entire training. No scheduled sampling is used. The vanilla shallow fusion is used without the “look-ahead” technique. The results, along with several baseline systems, are demonstrated and compared in Table 5,

max width= dev test clean other clean other Zeghidour et al. [57] 3.1 10.0 3.3 10.5 Hannun at al. [19] 3.0 8.9 3.3 9.8 Park et al. [38] (w/o SpecAugment) - - 3.2 9.8 Lüscher et al. [33] 2.6 8.4 2.8 9.3 Espresso LSTM 3.9 11.7 4.0 12.1  +Subword LM 3.5 9.1 3.4 9.7   +Improved Coverage 3.0 9.0 3.2 9.4   +EOS Threshold 2.9 8.7 2.8 9.1

Table 5: WERs (%) on the LibriSpeech dev and test sets.

We can see that both the improved coverage or EOS threshold help in this setup as well, where actually deletion error reductions contribute mostly. In addition, we achieve state-of-the-art results on end-to-end models for LibriSpeech without any data augmentation.

Switchboard The vocabulary consists of 1,000 subword units segmented by SentencePiece101010 It includes additional special tokens [laughter], [noise], and [vocalized-noise]. trained on both Switchboard and Fisher transcripts. As there is no official validation set, we hold out the same 4,000-example subset of the training data as in Kaldi for validation. We apply scheduled sampling starting at epoch 6 with probability from 0.9 to 0.6. Uniform smoothing is used with .

The language model achieves a validation perplexity of 17.3. No coverage is used during decoding. The results of our current system and 3 other competitive end-to-end baselines are shown in Table 6. Again, we obtain state-of-the-art results without SpecAugment. The coverage term or EOS threshold does not help on this dataset, and we suspect it is because its optimal LM fusion weight is not as large as those for the other two datasets, making less deletion errors.

max width= Switchboard CallHome Cui et al. [12] (w/ speed-pertubation) 12.0 23.1 Zeyer et al. [59] 11.0 23.1 Park et al. [38] (w/o SpecAugment) 10.9 19.4 Espresso LSTM 10.7 20.7  +Subword LM 9.2 19.1

Table 6: WERs (%) on the SWBD Hub5’00 evaluation set.

5 Training and Decoding Speed

In this section we compare Espresso and ESPnet on the training and decoding time with single GPU, using the WSJ dataset.

For fair comparison, we create architectures in Espresso (different from those in Section 4) that mimics the WSJ recipe in ESPnet as closely as possible. Data preparation and vocabulary building are identical. The neural architecture is mostly the same (e.g. number and dimension of LSTM layers), with a few minor exceptions: e.g. ESPnet’s use of location-based attention (which Espresso does not employ), pooling CNN layers (Espresso

uses strided CNN without pooling), and joint cross-entropy+CTC loss (

Espresso uses only cross-entropy loss).

Table 7 gives training wall time comparisons on both the external word-based language model and the encoder-decoder model, which are averaged over 20 epochs and 15 epochs respectively. Espresso is 17.9% faster than ESPnet on the language model training, and 16.0% faster on the encoder-decoder model training. We conjecture that the main reason of such speed gain for language model training is that in fairseq (and hence in Espresso) training examples are sorted based on input sequence lengths before batching (i.e., bucketing; ESPnet does not use it for language modeling), so that the average sequence length in batches is smaller.

max width= Training ASR Decoding (eval92) LM ASR w/o LM w/ look-ahead LM ESPnet 56min 36min 5min 21s 29min 16s Espresso 46min 31min 1min 27s 2min 44s Speedup 17.9% 16.0% 3.7 10.7

Table 7: Training (per epoch) and decoding wall time on WSJ.

A notable advantage of Espresso compared to ESPnet is the decoding speed. In order to have a more fair comparison, we enable GPU batch decoding in ESPnet [46], and make batch and beam sizes of the two systems the same. We measure the decoding time on the WSJ eval92 data set, which consists of 333 utterances. Table 7 shows that, without language model fusion, Espresso is 3.7 faster than ESPnet. With the look-ahead language model fusion, the speedup is even more prominent—more than 10 faster—mostly due to our parallelized implementation of the look-ahead language model fusion (cf. Section 3.1), as opposed to ESPnet, where LM scores are computed iteratively over examples within a minibatch.

6 Conclusion

In this paper we present Espresso, an open-source end-to-end ASR toolkit built on top of fairseq, an extensible neural machine translation toolkit. In addition to advantages inherited from fairseq, Espresso supports various other features for ASR that are seamlessly integrated with fairseq, including reading data in Kaldi format, and efficient parallelized language model-fused decoding. We also provide ASR recipes for WSJ, LibriSpeech, and Switchboard data sets, and achieve state-of-the-art performance among end-to-end systems. By sharing the underlying infrastructure with fairseq, we hope Espresso will facilitate future joint research in speech and natural language processing, especially in sequence transduction tasks such as speech translation and speech synthesis.


  • [1] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, et al. (2016) Deep speech 2 : end-to-end speech recognition in english and mandarin. In Proc. ICML, pp. 173–182. Cited by: Table 3, §4.
  • [2] K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny (2018) Building competitive direct acoustics-to-word models for english conversational speech recognition. In Proc. ICASSP, pp. 4759–4763. External Links: Document Cited by: §2.1.
  • [3] D. Bahdanau, K. Cho, and Y. Bengio (2015) Neural machine translation by jointly learning to align and translate. In Proc. ICLR, External Links: Link Cited by: §2.3, §2.3.
  • [4] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio (2016) End-to-end attention-based large vocabulary speech recognition. In Proc. ICASSP, pp. 4945–4949. Cited by: §3.2.
  • [5] M. K. Baskar, L. Burget, S. Watanabe, M. Karafiát, T. Hori, and J. H. Černocký (2019) Promising accurate prefix boosting for sequence-to-sequence ASR. In Proc. ICASSP, Vol. , pp. 5646–5650. External Links: Document Cited by: §2.4, Table 3, §4.
  • [6] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer (2015)

    Scheduled sampling for sequence prediction with recurrent neural networks

    In Proc. NeurIPS, pp. 1171–1179. External Links: Link Cited by: §2.4.
  • [7] Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proc. ICML, pp. 41–48. Cited by: §4.
  • [8] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals (2016) Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Proc. ICASSP, pp. 4960–4964. External Links: Link, Document Cited by: §1.
  • [9] J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio (2014) End-to-end continuous speech recognition using attention-based recurrent NN: first results. Deep Learning and Representation Learning Workshop, NeurIPS abs/1412.1602. External Links: Link, 1412.1602 Cited by: §1, §2.3.
  • [10] J. Chorowski and N. Jaitly (2017) Towards better decoding and language model integration in sequence to sequence models. In Proc. INTERSPEECH, pp. 523–527. External Links: Link Cited by: 3rd item, §3.2, §4.
  • [11] R. Collobert, C. Puhrsch, and G. Synnaeve (2016) Wav2Letter: an end-to-end convnet-based speech recognition system. CoRR abs/1609.03193. External Links: Link, 1609.03193 Cited by: §4.
  • [12] J. Cui, C. Weng, G. Wang, J. Wang, P. Wang, C. Yu, D. Su, and D. Yu (2018) Improving attention-based end-to-end ASR systems with sequence-based loss functions. In Proc. SLT, pp. 353–360. External Links: Document Cited by: Table 6.
  • [13] S. Ding and P. Koehn (2019)

    Parallelizable stack long short-term memory

    In Proc. the Third Workshop on Structured Prediction for NLP, pp. 1–6. External Links: Link Cited by: §3.1.
  • [14] J. Godfrey, E. Holliman, and J. McDaniel (1992) SWITCHBOARD: telephone speech corpus for research and development. In Proc. ICASSP, Vol. 1, pp. 517–520. Cited by: §1, §4.
  • [15] A. Graves, S. Fernández, and J. Schmidhuber (2005) Bidirectional LSTM networks for improved phoneme classification and recognition. In Proc. ICANN, pp. 799–804. External Links: Link, Document Cited by: §2.3.
  • [16] A. Graves and N. Jaitly (2014) Towards end-to-end speech recognition with recurrent neural networks. In Proc. ICML, pp. 1764–1772. Cited by: §1.
  • [17] Ç. Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio (2015) On using monolingual corpora in neural machine translation. CoRR abs/1503.03535. External Links: Link, 1503.03535 Cited by: §3.
  • [18] H. Hadian, H. Sameti, D. Povey, and S. Khudanpur (2018) End-to-end speech recognition using lattice-free MMI. In Proc. INTERSPEECH, pp. 12–16. External Links: Document Cited by: Table 1, Table 3, §4.
  • [19] A. Hannun, A. Lee, Q. Xu, and R. Collobert (2019) Sequence-to-sequence speech recognition with time-depth separable convolutions. In Proc. INTERSPEECH, pp. 3785–3789. Cited by: §1, §3.3, Table 5.
  • [20] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proc. CVPR, pp. 770–778. External Links: Link, Document Cited by: §2.3.
  • [21] T. Hori, J. Cho, and S. Watanabe (2018) End-to-end speech recognition with word-based RNN language models. In Proc. SLT, pp. 389–396. External Links: Link, Document Cited by: §1, 3rd item, §3.1, §3.1, §4.
  • [22] T. Hori, S. Watanabe, and J. R. Hershey (2017) Multi-level language modeling and decoding for open vocabulary end-to-end speech recognition. In Proc. ASRU, Vol. , pp. 287–293. Cited by: 2nd item.
  • [23] A. Kannan, Y. Wu, P. Nguyen, T. N. Sainath, Z. Chen, and R. Prabhavalkar (2018) An analysis of incorporating an external language model into a sequence-to-sequence model. In Proc. ICASSP, pp. 5824–5828. External Links: Link, Document Cited by: §3, §3.
  • [24] D. P. Kingma and J. Ba (2015) Adam: A method for stochastic optimization. In Proc. ICLR, External Links: Link Cited by: §4.
  • [25] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush (2017) OpenNMT: open-source toolkit for neural machine translation. In Proc. ACL: System Demonstrations, pp. 67–72. External Links: Link, Document Cited by: §1.
  • [26] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur (2015) Audio augmentation for speech recognition. In Proc. INTERSPEECH, pp. 3586–3589. External Links: Link Cited by: §4.
  • [27] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, et al. (2007) Moses: open source toolkit for statistical machine translation. In Proc. ACL: Demo and Poster Sessions, pp. 177–180. Cited by: §1.
  • [28] O. Kuchaiev, B. Ginsburg, I. Gitman, V. Lavrukhin, C. Case, and P. Micikevicius (2018) OpenSeq2Seq: extensible toolkit for distributed and mixed precision training of sequence-to-sequence models. In Proc. Workshop for NLP Open Source Software (NLP-OSS), pp. 41–46. External Links: Link Cited by: Table 1, §1.
  • [29] T. Kudo (2018) Subword regularization: improving neural network translation models with multiple subword candidates. In Proc. ACL, pp. 66–75. External Links: Link Cited by: §2.1, 1st item.
  • [30] Z. Li, C. Callison-Burch, C. Dyer, J. Ganitkevitch, S. Khudanpur, L. Schwartz, W. N. Thornton, J. Weese, and O. F. Zaidan (2009) Joshua: an open source toolkit for parsing-based machine translation. In Proc. the Fourth Workshop on Statistical Machine Translation, pp. 135–139. Cited by: §1.
  • [31] T. Likhomanenko, G. Synnaeve, and R. Collobert (2019) Who needs words? lexicon-free speech recognition. In Proc. INTERSPEECH, pp. 3915–3919. Cited by: Table 3, §4.
  • [32] T. Luong, H. Pham, and C. D. Manning (2015) Effective approaches to attention-based neural machine translation. In Proc. EMNLP, pp. 1412–1421. External Links: Link Cited by: §2.3, §2.3.
  • [33] C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney (2019) RWTH ASR systems for LibriSpeech: hybrid vs attention. In Proc. INTERSPEECH, pp. 231–235. Cited by: Table 5.
  • [34] Y. Miao, M. Gowayyed, and F. Metze (2015) EESEN: end-to-end speech recognition using deep RNN models and wfst-based decoding. In Proc. ASRU, pp. 167–174. Cited by: Table 1.
  • [35] R. Müller, S. Kornblith, and G. E. Hinton (2019) When does label smoothing help?. CoRR abs/1906.02629. External Links: Link Cited by: §2.4.
  • [36] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli (2019) Fairseq: A fast, extensible toolkit for sequence modeling. In Proc. NAACL-HLT: Demonstrations, pp. 48–53. External Links: Link Cited by: §1.
  • [37] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) Librispeech: an ASR corpus based on public domain audio books. In Proc. ICASSP, pp. 5206–5210. External Links: Link, Document Cited by: §1, §4.
  • [38] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le (2019) SpecAugment: a simple data augmentation method for automatic speech recognition. In Proc. INTERSPEECH, pp. 2613–2617. Cited by: §2.4, Table 5, Table 6, §4.
  • [39] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer (2017) Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop, Cited by: §1.
  • [40] D. B. Paul and J. M. Baker (1992) The design for the wall street journal-based CSR corpus. In Proc. ICSLP, pp. 357–362. External Links: Link Cited by: §1, §4.
  • [41] G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. E. Hinton (2017) Regularizing neural networks by penalizing confident output distributions. In Proc. ICLR, Workshop Track, External Links: Link Cited by: 2nd item.
  • [42] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al. (2011) The kaldi speech recognition toolkit. In Proc. ASRU, Cited by: §1.
  • [43] D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur (2016) Purely sequence-trained neural networks for ASR based on lattice-free MMI. In Proc. INTERSPEECH, pp. 2751–2755. Cited by: §4.
  • [44] V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert (2019) Wav2Letter++: A fast open-source speech recognition system. In Proc. ICASSP, pp. 6460–6464. Cited by: Table 1.
  • [45] O. Press and L. Wolf (2017) Using the output embedding to improve language models. In Proc. EACL, pp. 157–163. External Links: Link Cited by: §4.
  • [46] H. Seki, T. Hori, S. Watanabe, N. Moritz, and J. L. Roux (2019) Vectorized beam search for ctc-attention-based speech recognition. In Proc. INTERSPEECH, pp. 3825–3829. Cited by: §5.
  • [47] R. Sennrich, B. Haddow, and A. Birch (2016) Neural machine translation of rare words with subword units. In Proc. ACL, pp. 1715–1725. External Links: Link Cited by: §2.1, 1st item.
  • [48] J. Shen, P. Nguyen, Y. Wu, Z. Chen, et al. (2019) Lingvo: a modular and scalable framework for sequence-to-sequence modeling. CoRR abs/1902.08295. External Links: Link, 1902.08295 Cited by: Table 1.
  • [49] H. Soltau, H. Liao, and H. Sak (2017) Neural speech recognizer: acoustic-to-word LSTM model for large vocabulary speech recognition. In Proc. of INTERSPEECH, pp. 3707–3711. Cited by: §2.1.
  • [50] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016)

    Rethinking the inception architecture for computer vision

    In Proc. CVPR, pp. 2818–2826. External Links: Link Cited by: 1st item, §2.4.
  • [51] S. Tokui, K. Oono, S. Hido, and J. Clayton (2015) Chainer: a next-generation open source framework for deep learning. In

    Proc. Workshop on Machine Learning Systems (LearningSys) in NeurIPS

    External Links: Link Cited by: §1.
  • [52] S. Toshniwal, A. Kannan, C. Chiu, Y. Wu, T. N. Sainath, and K. Livescu (2018) A comparison of techniques for language model integration in encoder-decoder speech recognition. In Proc. SLT, Vol. , pp. 369–375. Cited by: §3.
  • [53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Proc. NeurIPS, pp. 6000–6010. External Links: Link Cited by: §2.3.
  • [54] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai (2018) ESPnet: end-to-end speech processing toolkit. In Proc. INTERSPEECH, pp. 2207–2211. External Links: Document, Link Cited by: Table 1, §1, §2.3, §3.1.
  • [55] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al. (2016) Google’s neural machine translation system: bridging the gap between human and machine translation. CoRR abs/1609.08144. External Links: Link, 1609.08144 Cited by: §2.3.
  • [56] D. Yu and L. Deng (2014) Automatic speech recognition: a deep learning approach. Springer Publishing Company, Incorporated. External Links: ISBN 1447157788, 9781447157786 Cited by: §1.
  • [57] N. Zeghidour, Q. Xu, V. Liptchinsky, N. Usunier, G. Synnaeve, and R. Collobert (2018) Fully convolutional speech recognition. CoRR abs/1812.06864. External Links: Link, 1812.06864 Cited by: Table 3, Table 5, §4.
  • [58] A. Zeyer, T. Alkhouli, and H. Ney (2018) RETURNN as a generic flexible neural toolkit with application to translation and speech recognition. In Proc. ACL, System Demonstrations, pp. 128–133. Cited by: Table 1.
  • [59] A. Zeyer, A. Merboldt, R. Schlüter, and H. Ney (2018)

    A comprehensive analysis on attention models

    In Proc. IRASL Workshop, NeurIPS, Cited by: Table 6.
  • [60] Y. Zhang, W. Chan, and N. Jaitly (2017) Very deep convolutional networks for end-to-end speech recognition. In Proc. ICASSP, pp. 4845–4849. Cited by: §2.3.