1 Introduction
Over the past few years, neural networks has been widely used in some domains, such as large vocabulary continuous speech recognition (LVCSR) [1, 2, 3], image recognition [4, 5]
and neural machine translation
[6]. Fullyconnected feedforward deep neural networks (DNNs) and Recurrent neural networks (RNNs), especially long shortterm memory (LSTM) RNNs have shown the effective performance for LVCSR [7, 8, 9, 10]. It is significant that training with larger dataset could improve recognition accuracy. As a matter of fact, larger dataset does mean more training samples and more model parameters, and it is high time consumption to train neural networks with only one computing unit. Therefore, parallel training with multiGPUs is essential, but it leads to slower convergence. For multiGPUs training, the key problem is how to accelerate convergence and get further improvement.Minibatch based stochastic gradient descent (SGD) is the most prevalent method in neural network training procedure. Several methods are proposed based on it, and achieving encouraging performance for parallel training. Asynchronous SGD is a successful attempt
[11, 12], and it is shown that parallel training with asynchronous SGD can many times speedup without lowering the accuracy. Besides, synchronous SGD is another positive effort, where the parameter server waits for every workers to finish their computation and send their local models to it, and then it sends updated model back to all workers [13]. Synchronous SGD converges well in parallel training with data parallelism, and is also easy to be implemented.Model averaging is a method for largescale parallel training, which the final model is averaged from all parameters of separated models [14, 15]. Compared with single GPU training, it achieves linear speedup, but the accuracy decreases. Moreover, blockwise modelupdating filter (BMUF) provides another linear speedup approach with multiGPUs on the basis of model averaging. It can achieve improvement or nodegradation of recognition performance compared with minibatch SGD on single GPU [16].
It is demonstrated that the performance of moving average of the parameters obtained by SGD is as good as that of the parameters which minimize the empirical cost, and moving average parameters can be used as the estimator of them, if there are enough training samples
[17]. One pass learning is then proposed, and it is the combination of learning rate schedule and averaged SGD using moving average [18]. When the moving average model outperforms the model aggregated with model averaging, the moving average model is broadcasted to update local workers. Since the learning rate of one pass learning is difficult to be adjusted, it is challenging to train different models in different domains.In this paper, we propose a new approach which applies exponential moving average (EMA) directly in largescale synchronousbased parallel training. It is a kind of noninterference method that the EMA model is not broadcasted, after the parameters of each worker are synchronized. It is applied as the final model of the training. The exponential moving average method in parallel training will be described in Section 2. Neural network models are successfully trained for LVCSR, using this method. The experiments and results are presented in Section 3, followed by the conclusion in Section 4.
2 Exponential Moving Average Model
In recognition applications, the parameters of neural network is trained for classification. It’s also an optimization problem:
Where is the number of data points, is the input data and correspondent target,
is the loss function, and
denotes the network. Let be the parameters that minimize the empirical cost. Large scale recognition training needs to deal with the optimization problem with billions of training data, and makes it hard to find the. SGD and its variants are presented promising learning results for large scale optimization problem, and become the most popular methods of deep learning.
2.1 Model averaging and blockwise model updating filter
In order to reduce the time cost of training, data parallelism is implemented. The full training dataset is partitioned into splits without overlapping, and they are distributed to GPUs.
Each GPU optimizes local model in parallel with one split of training dataset. After a minibatch training, the global model is needed to update, and it is computed with model averaging or BMUF, and consequently broadcasted to GPUs to update their local models. For model averaging method, all local models are synchronized and averaged, and then aggregated model is sent back to GPUs [14, 15]. For BMUF method, the global model is employed, instead of in model averaging method. The synchronization and updating process of in BMUF as follows:
Where denotes model update, and is the globalmodel update. There are two parameters in BMUF, block momentum , and block learning rate . Then, the global model is updated as
Consequently, is broadcasted to all GPUs to update their local models.
It is worth noting that when block momentum and block learning rate are set as 0 and 1, BMUF becomes model averaging. We treat model averaging and BMUF as model averaging based methods.
2.2 Moving Average and Exponential Moving Average
Averaged SGD is proposed to further accelerate the convergence speed of SGD. Averaged SGD leverages the moving average (MA) as the estimator of [17]:
Where is computed by model averaging or BMUF. It is shown that can well converge to , with the large enough training dataset in single GPU training. It can be considered as a noninterference strategy that does not participate the main optimization process, and only takes effect after the end of entire optimization. However, for the parallel training implementation, each is computed by model averaging and BMUF with multiple models, and moving average model does not well converge, compared with single GPU training.
Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially nodegradation implementation of BMUF. But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as
The weight of each is equal in moving average method regardless the effect of temporal order. But closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final . As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero. After moving average based methods, the EMA parameters are updated recursively as
Here represents the degree of weight decrease, and called exponential updating rate. EMA is also a noninterference training strategy that is implemented easily, as the updated model is not broadcasted. Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly.
3 Experiments and Results
3.1 Training Data
In order to present the performance of our proposed method, we trained acoustic model for LVCSR. A large quantity of labeled data is needed for training a more accurate acoustic model. We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China. The dataset is created from anonymous online users’ search queries in Mandarin, and all audio file’s sampling rate is 16kHz, recorded by mobile phones. This dataset consists of many different conditions, such as diverse noise even low signaltonoise, babble, dialects, accents, hesitation and so on. The dataset is divided into training set, validation set and test set, and the quantity of them is shown in Table 1. The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously. The overfitting can be prevented in time, if there is a apparent gap between the frame error rate (FER) of training and validation set.
Dataset  Hours 

Training set  16150 
Validation set  850 
Test set  10 
Total  17010 
3.2 Experimental setup
LSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its longrange dependencies more accurately for temporal sequence conditions [19, 20]. Shenma voice search is a streaming service that intermediate recognition results displayed while users are still speaking. So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one. Thus, the parallel training procedure is unidirectional LSTMbased.
A 28dimensional filter bank feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network. The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a fullconnection layer. The out layer is a softmax layer with 11088 hidden markov model (HMM) tiedstates as output classes, the loss function is crossentropy (CE). The performance metric of the system in Mandarin is reported with character error rate (CER). The alignment of framelevel ground truth is obtained by GMMHMM system. Minibatched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs. The block learning rate and block momentum of BMUF are set as 1 and 0.9. 5gram language model is leveraged in decoder, and the vocabulary size is as large as 760000.
EMA method is proposed for parallel training problem. In our training system, it is employed on the MPIbased HPC cluster where 8 GPUs are used to train neural network models. Each GPU processes nonoverlap subset split from the entire large scale dataset in parallel.
Local models from distributed workers synchronize with each other in decentralized way. In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers. Computing resource of workers is wasted until aggregation of the parameter server done. Decentralized method makes full use of computing resource. There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers. Local model of th worker in workers cluster is split to pieces , and send to corresponding worker. In the aggregation phase, th worker computed splits of model and send updated model back to workers. As a result, all workers participate in aggregation and no computing resource is dissipated. It is significant to promote training efficiency, when the size of neural network model is too large. The EMA model is also updated additionally, but not broadcasting it.
Besides, frame stacking leads to reduce the computation and training time dramatically [21]. Frames are stacked so as that the network sees multiple frames at a time. The super frame after stacking is the input feature of the network and it contains abundant information. As a result, 3 frames are stacked without overlapping in the training procedure.
3.3 Results
The test set including about 9000 samples contains various real world conditions. It simulates the majority of user scenarios, and could well evaluate the performance of a trained model. BMUF based approach, which has no worse performance than the singleGPU training procedure, is the baseline of experiments. The results of MA and EMA methods on the basis of BMUF are presented, and we call them MAbased methods.
Since the EMA is a noninterference method, the performance can not be evaluated with the realtime FER. Therefore, the FER on validation sets are computed after every epoch. In order to present the decoding performance of MAbased methods, we extract 4 temporary models from each epoch to visualize the degradation of CER. FER curves of LSTM models trained with BMUF, MA and EMA methods are shown in Figure 1. It is significant that the frame accuracy of MAbased methods are higher than those of BMUF. Frame accuracies between MAbased methods have slight difference. Though the EMA only perform better than MA slightly on FER, there is an obvious difference between the CER of EMA and MA, as shown in Figure 2 which illustrates CER curves of different models after decoding. It demonstrates that decoding result of EMA is always much better than that of BMUF, but that of MA fluctuates greatly, and even higher than that of BMUF sometimes. From the Table 2 which shows the CER of final models trained from three methods, the superiority of EMA over the others can be also observed. EMA method achieves about relative 3.9% CER reduction on test set, while MA method only achieves relative 2.1% CER reduction.
Moreover, the CER of final DNN models with 8 layers are also presented in Table 2, and the CER of EMA method decreases relative 8.4% compared with baseline. Therefore, more accuracy models are trained with largescale parallel training using EMA method, and it is more stable than MA method.
Methods  CER (%)  

LSTM  DNN  
BMUF  4.37  7.52 
BMUF + MA  4.26  7.14 
BMUF + EMA  4.20  6.89 
4 Conclusion
The exponential moving average method is proposed in this paper for multiGPUs cluster parallel training with almost linear speedup. It is demonstrated that unidirectional LSTM and DNN models trained with EMA method have better decoding results than that of BMUF and traditional moving average methods for large vocabulary continues speech recognition in Mandarin. Our future work includes 1) Employing this method to CNN, Connectionist Temporal Classification (CTC), attentionbased neural networks and other hybrid deep neural network architecture; 2) Extending this method from framewise discriminative training to sequence discriminative training such as maximum mutual information (MMI) and segmental Minimum BayesRisk (sMBR); 3) Develop more approaches for parallel training with better performance.
References
 [1] Alex Graves and Navdeep Jaitly, “Towards endtoend speech recognition with recurrent neural networks.,” in ICML, 2014, vol. 14, pp. 1764–1772.
 [2] Alex Graves, Navdeep Jaitly, and Abdelrahman Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 273–278.
 [3] Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, et al., “Deep speech 2: Endtoend speech recognition in english and mandarin,” arXiv preprint arXiv:1512.02595, 2015.

[4]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,
“Imagenet classification with deep convolutional neural networks,”
in Advances in neural information processing systems, 2012, pp. 1097–1105. 
[5]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
“Deep residual learning for image recognition,”
in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, 2016, pp. 770–778.  [6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014.
 [7] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012.
 [8] Alex Graves, Abdelrahman Mohamed, and Geoffrey Hinton, “Speech recognition with deep recurrent neural networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on. IEEE, 2013, pp. 6645–6649.
 [9] Hasim Sak, Andrew W Senior, and Françoise Beaufays, “Long shortterm memory recurrent neural network architectures for large scale acoustic modeling.,” in Interspeech, 2014, pp. 338–342.
 [10] Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yaco, Sanjeev Khudanpur, and James Glass, “Highway long shortterm memory rnns for distant speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 5755–5759.
 [11] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al., “Large scale distributed deep networks,” in Advances in neural information processing systems, 2012, pp. 1223–1231.
 [12] Shanshan Zhang, Ce Zhang, Zhao You, Rong Zheng, and Bo Xu, “Asynchronous stochastic gradient descent for dnn training,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 6660–6663.
 [13] Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Jozefowicz, “Revisiting distributed synchronous sgd,” arXiv preprint arXiv:1604.00981, 2016.

[14]
Ryan McDonald, Keith Hall, and Gideon Mann,
“Distributed training strategies for the structured perceptron,”
in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2010, pp. 456–464.  [15] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola, “Parallelized stochastic gradient descent,” in Advances in neural information processing systems, 2010, pp. 2595–2603.
 [16] Kai Chen and Qiang Huo, “Scalable training of deep learning machines by incremental block training with intrablock parallel optimization and blockwise modelupdate filtering,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 5880–5884.
 [17] Boris T Polyak and Anatoli B Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM Journal on Control and Optimization, vol. 30, no. 4, pp. 838–855, 1992.
 [18] Wei Xu, “Towards optimal one pass large scale learning with averaged stochastic gradient descent,” arXiv preprint arXiv:1107.2490, 2011.
 [19] Michiel Hermans and Benjamin Schrauwen, “Training and analysing deep recurrent neural networks,” in Advances in Neural Information Processing Systems, 2013, pp. 190–198.
 [20] Haşim Sak, Félix de Chaumont Quitry, Tara Sainath, Kanishka Rao, et al., “Acoustic modelling with cdctcsmbr lstm rnns,” in Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on. IEEE, 2015, pp. 604–609.
 [21] Haşim Sak, Andrew Senior, Kanishka Rao, and Françoise Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” arXiv preprint arXiv:1507.06947, 2015.
Comments
There are no comments yet.