Noise Robust IOA/CAS Speech Separation and Recognition System For The Third 'CHIME' Challenge

by   Xiaofei Wang, et al.

This paper presents the contribution to the third 'CHiME' speech separation and recognition challenge including both front-end signal processing and back-end speech recognition. In the front-end, Multi-channel Wiener filter (MWF) is designed to achieve background noise reduction. Different from traditional MWF, optimized parameter for the tradeoff between noise reduction and target signal distortion is built according to the desired noise reduction level. In the back-end, several techniques are taken advantage to improve the noisy Automatic Speech Recognition (ASR) performance including Deep Neural Network (DNN), Convolutional Neural Network (CNN) and Long short-term memory (LSTM) using medium vocabulary, Lattice rescoring with a big vocabulary language model finite state transducer, and ROVER scheme. Experimental results show the proposed system combining front-end and back-end is effective to improve the ASR performance.



page 1

page 2

page 3

page 4


State-of-the-art Speech Recognition using EEG and Towards Decoding of Speech Spectrum From EEG

In this paper we first demonstrate continuous noisy speech recognition u...

Rank-1 Constrained Multichannel Wiener Filter for Speech Recognition in Noisy Environments

Multichannel linear filters, such as the Multichannel Wiener Filter (MWF...

On The Compensation Between Magnitude and Phase in Speech Separation

Deep neural network (DNN) based end-to-end optimization in the complex t...

LCANet: End-to-End Lipreading with Cascaded Attention-CTC

Machine lipreading is a special type of automatic speech recognition (AS...

Investigation of Independent Monaural Front-End Processing for Robust ASR without Retraining and Joint-Training

In recent years, monaural speech separation has been formulated as a sup...

Multi-channel Multi-frame ADL-MVDR for Target Speech Separation

Many purely neural network based speech separation approaches have been ...

Optimizing Speech Recognition For The Edge

While most deployed speech recognition systems today still run on server...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Automatic Speech Recognition (ASR) has been applied to many human-computer interaction systems, such as tablet computer, smartphones, personal computers and televisions. Meanwhile, robust ASR in noisy environments is paid more attention due to its applicable value. The 3rd ’CHiME’ speech separation and recognition challenge is such a platform for testing the recognition rate of noisy speech in complex environments [1]. Our contributions to CHiME are separated into two parts: front-end techniques and back-end techniques.

It is well known that a lot of front-end techniques aim at extracting clean desired speech signals. Among them, multi-channel system is proved effective to improve the front-end performance in noisy and reverberant environment so that it attracts more attention in consideration of better balance between noise reduction and speech distortion. As is known to all, more noise reduction doesn’t mean more clean desired speech. Speech distortion brought by artifacts affects ASR performance severely. Therefore, taking speech distortion into account in the multi-channel optimization criterion, multi-channel wiener filter (WMF) technique has been proposed to estimate the desired speech component in noisy environment

[2]. The technique is generalized as speech distortion weighted MWF (SDW-MWF). The tradeoff between noise reduction and speech distortion is taken into consideration. In principle, it is desired to have less noise reduction in speech dominant segments and more noise reduction otherwise. From this motivation, we improve the SDW-MWF by focusing on the tradeoff parameter optimization from the perspective of desired noise reduction control technique.

Recently, acoustic modelling based on the Deep Neural Networks (DNNs) has gained popularity with the consistent improvement in recognition performance over earlier Neural Network based front-ends (e.g.[3]

). DNNs are either deployed as the front-end for standard Hidden Markov Model based on Gaussian Mixture Models (HMM-GMMs), or in a hybrid form to directly estimate state level posteriors. As noted in several publications

[4, 5, 6, 7]

, DNNs show general word error rate (WER) improvements on the order of 10-30% relative across a variety of small and large vocabulary tasks when compared with HMM-GMMs built on classic features. A DNN is a conventional Multi-Layer Perceptron (MLP) with many internal or hidden layers. Convolutional Neural Networks (CNNs) are an alternative type of neural network that can be used to reduce spectral variations and model spectral correlations which exist in signals. CNNs are a more effective model for speech compared to DNNs


. Besides, Long Short-Term Memory (LSTM) is also a specific recurrent neural network (RNN) architecture that was designed to model temporal sequences and their long-range dependencies more accurately than conventional RNNs. LSTM are also proved more effective than DNNs and conventional RNNs for acoustic modeling

[9, 10]. In this paper, we take advantage of these techniques for acoustic modeling and make a combination of them to achieve a better ASR performance [11].

This paper is organized as follows. In section 2, 3, we describe the front-end and back-end of the proposed system. In section 4, we carry out ASR experiments and list the results with analysis. At last, we draw a conclusion in section 5.

2 Speech enhancement front-end

In order to suppress background noise, multichannel wiener filter (MWF) is introduced to the multi-microphone set-up [2]. Since MWF does not require transfer functions between a target speaker and microphones, it is suitable for the CHiME3 task. Taking speech distortion into account in its optimization criterion, MWF is generalized as speech distortion weighted multichannel wiener filter (SDW-MWF), which provides a tradeoff between speech distortion and noise reduction [12, 13, 14]. In this work, a tradeoff parameter optimized method based on SDW-MWF is used.

Considering an array of microphones. Let ,

denote the short-time Fourier transform (STFT) domain notation of

-th microphone signal at frequency index and frame index , the received signals are given as


where ,,, are respectively the STFT domain expression of the source signal , the transfer function from the source to the -th microphone , the target signal and noise signal at microphone .

To find an optimal estimate of the target signal, the designed SDW-MWF criterion is [13, 15]


where is the target signal at the first microphone,

is the received signal vector defined as

and , , , are defined similarly, among which represents the linear filter given by . Here operators and represent the transposition and Hermitian transpose operation respectively. Apparently, a larger value of emphasize more on noise reduction. Variables and are omitted here for simplicity. The solution to SDW-MWF can be obtained as


where is a -dimensional vector corresponds to the first microphone (channel 1 of the 6-microphone array), and are the correlation matrices of clean speech signal and noise signal, respectively.

Using a fixed parameter , the reduced residual noise level generally achieved at the expense of increased speech distortion. In our work, we compute the parameter according to desired noise reduction level.



denotes the imput signal-to-noise ratio (SNR) of the first microphone,

is a noise reduction control factor defined as , represents the noise power at the first microphone, and represents desired residual noise level. Apparently, when the background noise level is relatively high or the input SNR is relatively low, the optimized parameter will emphasize more on noise reduction, which is reasonable. In this work, the noise power and noise covariance matrix for each frequency bin are computed from the initial and final 10 frames of each utterance.

3 Back-End Description

3.1 Acousitic modeling with neural network

Figure 1: Back-end description

Fig.1 demonstrates the back-end description including the techniques we used of the proposed system.

The GMM baseline includes the standard triphone based acoustic models with various feature transformations including linear discriminant analysis (LDA), maximum likelihood linear transformation (MLLT), and feature space maximum likelihood linear regression (fMLLR) with speaker adaptive training (SAT).

The DNN baseline provides the state-of-the-art ASR performance. It is based on the Kaldi recipe for Track 2 of the 2nd CHiME Challenge [16]

. The DNN is trained using the standard procedure (pre-training using restricted Boltzmann machine, cross entropy training, and sequence discriminative training). This baseline requires relatively massive computational resources (GPUs for the DNN training and many CPUs for lattice generation).

We start DNN training based on scripts of baseline system. We use 7 hidden layers and 2048 nodes for each hidden layer. The features for the DNN training are 40-dimensional filter-bank and its delta, delta-delta features. A context window of 11 frames (5+1+5) is used so that the dimension of the input layer for DNN is 403

11. Cepstral Mean and Variance Normalization (CMVN) is applied and proves to be useful. The DNN output layer size is the same as the GMM-HMM, which is 2024. The DNN is trained using the standard procedure like baseline system.

The CNN uses fbank+pitch features and contains two convolutional hidden layers and a max-pooling layer. The input feature vector (not including pitch) is divided into 40 bands. The corresponding dimension of the 11 consecutive feature frames are arranged in each band, together with their derivatives. So that the input dimension of the CNN is 43

311. The first set of convolutional filters are applied to 8 consecutive bands and generate 128 feature mappings. We then apply max-pooling across 3 bands to generate 11 bands. The second set of convolutional filters are applied to 4 consecutive bands and generate 256 feature mappings. Four fully-connected hidden layers of 1024 nodes are arranged after the convolutional layers. The total number of parameters for the CNN is 7.7M.

The LSTM network used in this paper is a two layer LSTM RNN, where each LSTM layer has 1024 memory cells and a dimensionality reducing recurrent projection layer of 200 linear units [9, 10].

In our experiments, we use an official trigram language model (LM) on the initial decoding pass and use a 5-order LM for lattice rescoring in a second pass. The official trigram LM has 5k vocabularies. The 5-order LM is trained using official training data only, but has vocabularies up to 12k.

3.2 Combination of different systems

To combine these multiple speech recognition outputs into a single one, we employ ROVER at the decision level [11] in the final step. The fusion enables us to achieve a lower error rate than any of the individual systems alone. In this paper, NIST scoring toolkit (SCTK,version 1.3) is used as a rover tool to combine the different results. It takes N input files and does an N-way dynamic programming (DP) alignment on those files. The output is a voted output depending the maximum confidence score.

4 Experiments and Results

The experiments are all carried out following the instructions of CHiME challenge. In this section, we list the ASR improvement step by step according to each technique we used resulting in the final WER of the test set provided by CHiME challenge. Table.1 gives the GMM and DNN baselines ’CHiME’ provided, Table.2 shows the ASR results by the proposed system and Table.3 shows the ASR results under each scenario including the bus (BUS), cafe (CAF), pedestrian area (PED), and street junction (STR) according to the best system after ROVER.

Model Test Data Training Data Dev. Set Test Set
Real Sim. Real Sim.
noisy clean 55.65 50.25 79.84 63.30
GMM noisy 18.70 18.71 33.23 21.59
MVDR clean 41.88 21.72 78.12 25.63
MVDR 20.55 9.79 37.36 10.59
DNN+sMBR noisy noisy 16.13 14.30 33.43 21.51
DNN+sMBR MVDR MVDR 17.72 8.17 33.76 11.19
Table 1: WER Baselines from the 3rd CHiME challenge.
Model Test Data Training Data Dev. Set Test Set
Real Sim. Real Sim.
GMM SDW-MWF Clean 30.23 29.75 53.43 41.58
SDW-MWF 13.16 14.11 23.19 18.65
GMM SDW-MWF Random SNR+SDW-MWF 13.01 13.95 22.07 17.57
GMM+Rescore 11.61 12.37 20.35 15.7
DNN+sMBR SDW-MWF Random SNR+SDW-MWF 9.95 10.03 18.4 12.98
DNN+sMBR+Rescore 8.48 9.01 15.3 11.29
CNN+sMBR SDW-MWF Random SNR+SDW-MWF 9.52 9.64 17.87 12.64
CNN+sMBR+Rescore 8.51 8.77 16.37 11.55
LSTM SDW-MWF Random SNR+SDW-MWF 10.81 11.18 18.96 14.1
LSTM+Rescore 9.44 9.71 16.45 12.48
ROVER SDW-MWF Random SNR+SDW-MWF 7.29 7.68 13.2 9.71
Table 2: WERs of proposed system.

4.1 ASR performance of front-end speech enhancement

As mentioned above, front-end speech enhancement brings benefits to the ASR performance. Table.2 demonstrates that WER of real test data decreases from 37.36% to 23.19% by changing the speech enhancement method from MVDR (supplied by CHiME organizers [17]) to the proposed SDW-MWF under GMM acoustic model. If we randomize the SNR of training data from -6dB–6dB (denoted by Random SNR in Table.2) instead of the estimated SNR calculated from really recorded data for simulating training set, the WER decreases to 22.07%.

Under DNN+sMBR acoustic model, the WER decreases from 33.76% to 18.4% on test data using SDW-MWF and random SNR schemes. It is worthy mentioning that all the training data is enhanced to compensate the mismatch between the training data and test data.

4.2 Back-end ASR performance

The results of DNN model on the development and evaluation set are also given in Table 2. we can see that DNN get 16.63% relative WER reduction comparing with GMM system on the real data of the test set. Obviously, the improvement is not enough, then we tried to use several other NN topologies.

As it is shown in Table 2, the CNN acoustic models as it has shown superior performance over conventional DNN. The WER decreases from 18.4% to 17.87%. Table 2 shows that LSTM gets further improvement. 14.09% relative reduction was achieved comparing to GMM. After lattice rescoring, all of the systems get significantly improvement.

Finally the best ASR result was obtained by combining all the systems with lattice rescoring together. We achieve a final WER of 13.2% on the real data of the test set, resulting in a 60.9% relative reduction in WER compared to the result of 33.23% from the best GMM-baseline. Table.3 shows the detail ASR results under different recording scenarios.

The best single system is the DNN+sMBR using lattice rescoring shown by Table.2.

Environment Dev. Set Test Set
Real Sim. Real Sim.
BUS 8.88 6.77 17.74 7.4
CAF 7.08 9.94 11.75 10.95
PED 5.78 6.14 13.34 9.19
STR 7.4 7.89 9.96 11.32
Table 3: WERs of the best system under different environments.

5 Conclusion

A state-of-the-art ASR system is presented in this paper facing with the task of reducing the effects of noise under different real applicable scenarios using a 6-microphone array. Two aspects are stated separately. Front-end speech enhancement using SDW-MWF achieves considerable performance improvement. Back-end techniques including GMM, DNN, CNN and LSTM are investigated. The combination of the four systems with lattice rescoring has the best ASR performance on the develop and test set. we achieve a relative 60.9% WER reduction on the real data of the test data compared to the best baseline system.


  • [1] Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe, “The third ’chime’ speech separation and recognition challenge: Dataset, task and baselines,” in Submitted to IEEE 2015 Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2015.
  • [2] Simon Doclo and Marc Moonen, “Gsvd-based optimal filtering for multi-microphone speech enhancement,” in Microphone Arrays, pp. 111–132. Springer, 2001.
  • [3] F. Grezl, M. Karafiat, S. Kontar, and J. Cernocky, “Probabilistic and bottle-neck features for LVCSR of meetings,” in Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. IEEE International Conference on, 2007, vol. 4, pp. IV–757–IV–760.
  • [4] H. Larochelle, Y. Bengio, J. Louradour, and P. Lamblin, “Exploring strategies for training deep neural networks,” J. Mach. Learn. Res., vol. 10, pp. 1–40, June 2009.
  • [5] F. Seide, G. Li, and D. Yu, “Conversational speech transcription using context-dependent deep neural networks,” in INTERSPEECH, 2011, pp. 437–440.
  • [6] P. Swietojanski, A. Ghoshal, and S. Renals, “Hybrid acoustic models for distant and multichannel large vocabulary speech recognition,” in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, Dec 2013, pp. 285–290.
  • [7] Yulan Liu, Pengyuan Zhang, and Thomas Hain, “Using neural network front-ends on far field multiple microphones based speech recognition,” in ICASSP2014 - Speech and Language Processing (ICASSP2014 - SLTC), Florence, Italy, May 2014, pp. 5579–5583.
  • [8] T. N. Sainath, A. R. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for LVCSR,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, 2013, pp. 8614–8618.
  • [9] Hasim Sak, Andrew Senior, and Fran04oise Beaufays, “Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition,” Eprint Arxiv 1402, 2014.
  • [10] Hasim Sak, Andrew Senior, and Fran04oise Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” Interspeech, pp. 338 – 342, 2014.
  • [11] J. G. Fiscus, “A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER),” in IEEE Workshop on Automatic Speech Recognition and Understanding, 1997.
  • [12] Mehrez Souden, Jacob Benesty, and Sofiène Affes, “On optimal frequency-domain multichannel linear filtering for noise reduction,” Audio, Speech, and Language Processing, IEEE Transactions on, vol. 18, no. 2, pp. 260–276, 2010.
  • [13] Ann Spriet, Marc Moonen, and Jan Wouters, “Spatially pre-processed speech distortion weighted multi-channel wiener filtering for noise reduction,” Signal Processing, vol. 84, no. 12, pp. 2367–2387, 2004.
  • [14] Simon Doclo and Marc Moonen, “Gsvd-based optimal filtering for single and multimicrophone speech enhancement,” Signal Processing, IEEE Transactions on, vol. 50, no. 9, pp. 2230–2244, 2002.
  • [15] Simon Doclo, Ann Spriet, Jan Wouters, and Marc Moonen, “Frequency-domain criterion for the speech distortion weighted multichannel wiener filter for robust noise reduction,” Speech Communication, vol. 49, no. 7, pp. 636–656, 2007.
  • [16] Chao Weng, Dong Yu, Shigetaka Watanabe, and Biing-Hwang Fred Juang, “Recurrent deep neural networks for robust speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 5532–5536.
  • [17] Xavier Mestre, Miguel Lagunas, et al., “On diagonal loading for minimum variance beamformers,” in Signal Processing and Information Technology, 2003. ISSPIT 2003. Proceedings of the 3rd IEEE International Symposium on. IEEE, 2003, pp. 459–462.