Statistical language models are a crucial component of speech recognition, machine translation and information retrieval systems. In order to handle the data sparsity problem associated with traditional -gram language models (LMs), neural network language models (NNLMs) (Bengio et al., 2001)
represent the history context in a continuous vector space that can be learned towards error rate reduction by sharing data among similar contexts. Instead of using fixed number of words to represent context, recurrent neural network language models (RNNLMs)(Mikolov et al., 2010) use a recurrent hidden layer to represent longer and variable length histories. RNNLMs significantly outperform traditional -gram LMs, and are therefore becoming an increasingly popular choice for practitioners (Mikolov et al., 2010; Sundermeyer et al., 2013; Devlin et al., 2014).
Consider a standard RNNLM, depicted in Figure 1. The network has an input layer , a hidden layer (also called context layer or state) with a recurrent connection to itself, and an output layer . Typically, at time step the network is fed as input , where denotes the vocabulary size, and , the previous state. It produces a hidden state , where is the size of the hidden layer, which in turn is transformed to the output . Different layers are fully connected, with the weight matrices denoted by .
For language modeling applications, the input is a sparse vector of a 1-of-
(or one-hot) encoding with the element corresponding to the input wordbeing 1 and the rest of components of set to 0; the state of the network is a dense vector, summarizing the history context preceding the output word ; and the output is a dense vector, with the
-th element denoting the probability of the next word being, that is, , or more concisely, . The input to output transformation occurs via:
is the sigmoid activation function, andis the softmax function .
One can immediately see that if uses a 1-of- encoding, then the computations in equation (1) are relatively inexpensive (typically is of the order of a few thousand, and the computations are ), while the computations in equation (2) are expensive (typically is of the order of a million, and the computations are ). Similarly, back propagating the gradients from the output layer to the hidden layer is expensive. Consequently, the training times for some of the largest models reported in literature are of the order of weeks (Mikolov et al., 2011; Williams et al., 2015).
In this paper, we ask the following question: Can we design an approximate training scheme for RNNLM which will improve on the state of the art models, while using significantly less computational resources? Towards this end, we propose BlackOut an approximation algorithm to efficiently train massive RNNLMs with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a weighted sampling strategy which significantly reduces computation while improving stability, sample efficiency, and rate of convergence. We also establish close connections between BlackOut, importance sampling, and noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2012; Mnih & Teh, 2012), and demonstrate that BlackOut mitigates some of the limitations of both previous methods. Our experiments, on the recently released one billion word language modeling benchmark (Chelba et al., 2014), demonstrate scalability and accuracy of BlackOut; we outperform the state-of-the art, achieving the lowest perplexity scores on this dataset. Moreover, unlike other established methods which typically require GPUs or CPU clusters, we show that a carefully implemented version of BlackOut requires only 1-10 days on a single CPU machine to train a RNNLM with a million word vocabulary and billions of parameters on one billion words.
One way to understand BlackOut is to view it as an extension of the DropOut strategy (Srivastava et al., 2014) to the output layer, wherein we use a discriminative training loss and a weighted sampling scheme. The connection to DropOut is mainly from the way they operate in model training and model evaluation. Similar to DropOut, in BlackOut training a subset of output layer is sampled and trained at each training batch and when evaluating, the full network participates. Also, like DropOut, a regularization technique, our experiments show that the models trained by BlackOut are less prone to overfitting. A primary difference between them is that DropOut is routinely used at input and/or hidden layers of deep neural networks, while BlackOut only operates at output layer. We chose the name BlackOut in light of the similarities between our method and DropOut, and the complementary they offer to train deep neural networks.
2 BlackOut: A sampling-based approximation
We will primarily focus on estimation of the matrix . To simplify notation, in the sequel we will use to denote and to denote the -th row of . Moreover, let denote the dot product between two vectors. Given these notations, one can rewrite equation (2) as
RNNLMs with a softmax output layer are typically trained using cross-entropy as the loss function, which is equivalent to maximum likelihood (ML) estimation, that is, to find the model parameterwhich maximizes the log-likelihood of target word , given a history context :
whose gradient is given by
The gradient of log-likelihood is expensive to evaluate because (1) the cost of computing is and (2) the summation above takes time linear in the vocabulary size .
To alleviate the computational bottleneck of computing the gradient (5), we propose to use the following discriminative objective function for training RNNLM:
where is a set of indices of words drawn from the vocabulary, and . Typically, is a tiny fraction of , and in our experiments we use . To generate we will sample words from the vocabulary using an easy to sample distribution , and set in order to compute
Equation 6from negative samples . The first term in (6) corresponds to the traditional maximum likelihood training, and the second term explicitly pushes down the probability of negative samples in addition to the implicit shrinkage enforced by the denominator of (7). In our experiments, we found the discriminative training (6) outperforms the maximum likelihood training (the first term of Eq. 6) in all the cases, with varying degree of accuracy improvement depending on .
The weighted softmax function (7) can be considered as a stochastic version of the standard softmax (3) on a different base measure. While the standard softmax (3) uses a base measure which gives equal weights to all words, and has support over the entire vocabulary, the base measure used in (7) has support only on words: the target word and samples from . The noise portion of (7) has the motivation from the sampling scheme, and the term for target word is introduced mainly to balance the contributions from target word and noisy sample words.111It’s shown empirically in our experiments that setting in (7) hurts the accuracy significantly. Other justifications are discussed in Sec. 2.1 and Sec. 2.2, where we establish close connections between BlackOut, importance sampling, and noise contrastive estimation.
Due to the weighted sampling property of BlackOut, some words might be sampled multiple times according to the proposal distribution , and thus their indices may appear multiple times in . As is the target word, which is assumed to be included in computing (7), we therefore set explicitly.
Then taking derivatives with respect to , yields
By the chain rule of derivatives, we can propagate the errors backward to previous layers and compute the gradients with respect to the full model parameters. In contrast to Eq. 5, Eqs. 9 and 10 are much cheaper to evaluate as (1) the cost of computing is and (2) the summation takes , hence roughly a times of speed-up.
Next we turn our attention to the proposal distribution
. In the past, a uniform distribution or the unigram distribution have been advocated as promising candidates for sampling distributions(Bengio & Senécal, 2003; Jean et al., 2015; Bengio & Senécal, 2008; Mnih & Teh, 2012). As we will see in the experiments, neither one is suitable for a wide range of datasets, and we find that the power-raised unigram distribution of Mikolov et al. (2013) is very important in this context:
Note that is a generalization of uniform distribution (when ) and unigram distribution (when ). The rationale behind our choice is that by tuning
, one can interpolate smoothly between sampling popular words, as advocated by the unigram distribution, and sampling all words equally. The bestis typically dataset and/or problem dependent; in our experiments, we use a holdout set to find the best value of . It’s worth noting that this sampling strategy has been used by Mikolov et al. (2013) in a similar context of word embedding, while here we explore its effect in the language modeling applications.
After BlackOut training, we evaluate the predictive performance of RNNLM by perplexity. To calculate perplexity, we explicitly normalize the output distribution by using the exact softmax function (3). This is similar to DropOut (Srivastava et al., 2014), wherein a subset of network is sampled and trained at each training batch and when evaluating, the full network participates.
2.1 Connection to Importance Sampling
BlackOut has a close connection to importance sampling (IS). To see this, differentiating the logarithm of Eq. 7 with respect to model parameter , we have
Importance sampling has been applied to NNLMs with large output layers in previous works (Bengio & Senécal, 2003, 2008; Jean et al., 2015). However, either uniform distribution or unigram distribution is used for sampling and all aforementioned works exploit the maximum likelihood learning of model parameter . By contrast, BlackOut uses a discriminative training (6) and a power-raised unigram distribution for sampling; these two changes are important to mitigate some of limitations of IS-based approaches. While an IS-based approach with a uniform proposal distribution is very stable for training, it suffers from large bias due to the apparent divergence of the uniform distribution from the true data distribution
. On the other hand, a unigram-based IS estimate can make learning unstable due to the high variance(Bengio & Senécal, 2003, 2008). Using a power-raised unigram distribution entails a better trade-off between bias and variance, and thus strikes a better balance between these two extremes. In addition, as we will see from the experiments, the discriminative training of BlackOut speeds up the rate of convergence over the traditional maximum likelihood learning.
2.2 Connection to Noise Contrastive Estimation
The basic idea of NCE is to transform the density estimation problem to the problem of learning by comparison, e.g., estimating the parameters of a binary classifier that distinguishes samples from the data distribution from samples generated by a known noise distribution (Gutmann & Hyvärinen, 2012). In the language modeling setting, the data distribution will be the distribution of interest, and the noise distribution is often chosen from the ones that are easy to sample from and possibly close to the true data distribution (so that the classification problem isn’t trivial). While Mnih & Teh (2012) uses a context-independent (unigram) noise distribution , BlackOut can be formulated into the NCE framework by considering a context-dependent noise distribution , estimated from samples drawn from , by
which is a probability distribution function under the expectation thatsamples are drawn from : since and (See the proof in Appendix A).
Similar to Gutmann & Hyvärinen (2012), noise samples are assumed times more frequent than data samples so that data points are generated from a mixture of two distributions: and . Then the conditional probability of sample being generated from the data distribution is
which is exactly the weighted softmax function defined in (7). Note that due to the noise distribution proposed in Eq. 13, the expensive denominator (or the partition function ) of is canceled out, while in Mnih & Teh (2012) the partition function is either treated as a free parameter to be learned or approximated by a constant. Mnih & Teh (2012) recommended to set in the NCE training. However, from our experiments, setting often leads to sub-optimal solutions222Similarly, Chen et al. (2015) reported that setting gave them the best results. and different settings of sometimes incur numerical instability since the log-sum-exp trick333https://en.wikipedia.org/wiki/LogSumExp can not be used there to shift the scores of the output layer to a range that is amenable to the exponential function. BlackOut does not have this hyper-parameter to tune and the log-sum-exp trick still works for the weighted softmax function (7). Due to the discriminative training of NCE and BlackOut, they share the same objective function (6).
We shall emphasize that according to the theory of NCE, the samples should be sampled from the noise distribution . But in order to calculate , we need the samples drawn from beforehand. As an approximation, we use the same samples drawn from as the samples from , and only use the expression of in (13) to evaluate the noise density value required by Eq. 14. This approximation is accurate since as proved in Appendix A, and we find empirically that it performs much better (with improved stability) than using a unigram noise distribution as in Mnih & Teh (2012).
2.3 Related Work
Many approaches have been proposed to address the difficulty of training deep neural networks with large output spaces. In general, they can be categorized into four categories:
words as its leaves. It allows exponentially faster computation of word probabilities and their gradients, but the predictive performance of the resulting model is heavily dependent on the tree used, which is often constructed heuristically. Moreover, by relaxing the constraint of a binary structure,Le et al. (2011) introduces a structured output layer with an arbitrary tree structure constructed from word clustering. All these methods speed up both the model training and evaluation considerably.
Sampling-based approximations select at random or heuristically a small subset of the output layer and estimate gradient only from those samples. The use of importance sampling in Bengio & Senécal (2003, 2008); Jean et al. (2015), and the use of NCE (Gutmann & Hyvärinen, 2012) in Mnih & Teh (2012) all fall under this category, so does the more recent use of Locality Sensitive Hashing (LSH) techniques (Shrivastava & Li, 2014; Vijayanarasimhan et al., 2014) to select a subset of good samples. BlackOut, with close connections to importance sampling and NCE, also falls in this category. All these approaches only speed up the model training, while the model evaluation still remains computationally challenging.
Self normalization (Devlin et al., 2014) extends the cross-entropy loss function by explicitly encouraging the partition function of softmax to be as close to 1.0 as possible. Initially, this approach only speeds up the model evaluation and more recently it’s extended to facilitate the training as well with some theoretical guarantees (Andreas & Klein, 2014; Andreas et al., 2015).
Exact gradient on limited loss functions (Vincent et al., 2015) introduces an algorithmic approach to efficiently compute the exact loss, gradient update for the output weights in per training example instead of . Unfortunately, it only applies to a limited family of loss functions that includes squared error and spherical softmax, while the standard softmax isn’t included.
As discussed in the introduction, BlackOut also shares some similarity to DropOut (Srivastava et al., 2014). While DropOut is often applied to input and/or hidden layers of deep neural networks to avoid feature co-adaptation and overfitting by uniform sampling, BlackOut applies to a softmax output layer, uses a weighted sampling, and employs a discriminative training loss. We chose the name BlackOut in light of the similarities between our method and DropOut, and the complementary they offer to train deep neural networks.
3 Implementation and Further Speed-up
We implemented BlackOut on a standard machine with a dual-socket 28-core Intel®Xeon®444Intel and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Haswell CPU. To achieve high throughput, we train RNNLM with Back-Propagation Through Time (BPTT) (Rumelhart et al., 1988) with mini-batches (Chen et al., 2014)
. We use RMSProp(Hinton, 2012)
for learning rate scheduling and gradient clipping(Bengio et al., 2013) to avoid the gradient explosion issue of recurrent networks. We use the latest Intel MKL library (version 11.3.0) for SGEMM calls, which has improved support for tall-skinny matrix-matrix multiplications, which consume about 80% of the run-time of RNNLMs.
It is expensive to access and update large models with billions of parameters. Fortunately, due to the 1-of- encoding at input layer and the BlackOut sampling at output layer, the model update on and is sparse, i.e., only the model parameters corresponding to input/output words and the samples in are updated at each training batch. However, subnet updates have to be done carefully due to the dependency within RMSProp updating procedure. We therefore propose an approximated RMSProp that enables an efficient subnet update and thus speeds up the algorithm even further. Details can be found in Appendix C.
In our experiments, we first compare BlackOut, NCE and exact softmax (without any approximation) using a small dataset. We then evaluate the performance of BlackOut on the recently released one billion word language modeling benchmark (Chelba et al., 2014) with a vocabulary size of up to one million. We compare the performance of BlackOut on a standard CPU machine versus the state-of-the-arts reported in the literature that are achieved on GPUs or on clusters of CPU nodes. Our implementation and scripts are open sourced at https://github.com/IntelLabs/rnnlm.
Models are trained and evaluated on two different corpora: a small dataset provided by the RNNLM Toolkit555http://www.rnnlm.org/, and the recently released one billion word language modeling benchmark666https://code.google.com/p/1-billion-word-language-modeling-benchmark/, which is perhaps the largest public dataset in language modeling. The small dataset has 10,000 training sentences, with 71,350 words in total and 3,720 unique words; and the test perplexity is evaluated on 1,000 test sentences. The one billion word benchmark was constructed from a monolingual/English corpora; after all necessary preprocessing including de-duplication, normalization and tokenization, 30,301,028 sentences (about 0.8 billion words) are randomly selected for training, 6,075 sentences are randomly selected for test and the remaining 300,613 sentences are reserved for future development and can be used as holdout set.
4.1 Results On Small Dataset
We evaluate BlackOut, NCE and exact softmax (without any approximation) on the small dataset described above. This small dataset is used so that we can train the standard RNNLM algorithm with exact softmax within a reasonable time frame and hence to provide a baseline of expected perplexity. There are many other techniques involved in the training, such as RMSProp for learning rate scheduling (Hinton, 2012), subnet update (Appendix C), and mini-batch splicing (Chen et al., 2014), etc., which can affect the perplexity significantly. For a fair comparison, we use the same tricks and settings for all the algorithms, and only evaluate the impact of the different approximations (or no approximation) on the softmax output layer. Moreover, there are a few hyper-parameters that have strong impact on the predictive performance, including of the proposal distribution for BlackOut and NCE, and additionally for NCE, the partition function . We pay an equal amount of effort to tune these hyper-parameters for BlackOut and NCE on the validation set as number of samples increases.
Figure 2 shows the perplexity reduction as a function of number of samples under two different vocabulary settings: (a) a full vocabulary of 3,720 words, and (b) using the most frequent 2,065 words as vocabulary. The latter is a common approach used in practice to accelerate RNNLM computation by using RNNLM to predict only the most frequent words and handling the rest using an -gram model (Schwenk & Gauvain, 2005). We will see similar vocabulary settings when we evaluate BlackOut on the large scale one billion word benchmark.
As can be seen, when the size of the samples increases, in general both BlackOut and NCE improve their prediction accuracy under the two vocabulary settings, and even with only 2 samples both algorithms still converge to reasonable solutions. BlackOut can utilize samples much more effectively than NCE as manifested by the significantly lower perplexities achieved by BlackOut, especially when number of samples is small; Given about 20-50 samples, BlackOut and NCE reach similar perplexities as the exact softmax, which is expensive to train as it requires to evaluate all the words in the vocabularies. When the vocabulary size is 2,065, BlackOut achieves even better perplexity than that of the exact softmax. This is possible since BlackOut does stochastic sampling at each training example and uses the full softmax output layer in prediction; this is similar to DropOut that is routinely used in input layer and/or hidden layers of deep neural networks (Srivastava et al., 2014). As in DropOut, BlackOut has the benefit of regularization and avoids feature co-adaption and is possibly less prone to overfitting. To verify this hypothesis, we evaluate the perplexities achieved on the training set for different algorithms and provide the results in Figure 5 at Appendix B. As can been seen, the exact softmax indeed overfits to the training set and reaches lower training perplexities than NCE and BlackOut.
Next we compare the convergence rates of BlackOut and NCE when training the RNNLMs with 16 hidden units for a full vocabulary of 3,720 words. Figures 3 and 3 plot the learning curves of BlackOut and NCE when 10 samples or 50 samples are used in training, respectively. The figure shows that BlackOut enjoys a much faster convergence rate than NCE, especially when number of samples is small (Figure 3); but this advantage gets smaller when number of samples increases (Figure 3). We also observed similar behavior when we evaluated BlackOut and NCE on the large scale one billion word benchmark.
4.2 Results On One Billion Word Benchmark
We follow the experiments from Williams et al. (2015) and Le et al. (2015) and compare the performance of BlackOut with the state-of-the-art results provided by them. While we evaluated BlackOut on a dual-socket 28-core Intel®Xeon®Haswell machine, Williams et al. (2015) implemented RNNLM with the NCE approximation on NVIDIA GTX Titan GPUs, and Le et al. (2015) executed an array of recurrent networks, including deep RNN and LSTM, without approximation on a CPU cluster. Besides the time-to-solution comparison, these published results enable us to cross-check the predictive performance of BlackOut with another implementation of NCE or with other competitive network architectures.
4.2.1 When vocabulary size is 64K
Following the experiments in Williams et al. (2015), we evaluate the performance of BlackOut on a vocabulary of 64K most frequent words. This is similar to the scenario in Figure 2 where the most frequent words are kept in vocabulary and the rest rare words are mapped to a special <unk> token. We first study the importance of of the proposal distribution and the discriminative training (6) as proposed in BlackOut. As we discussed in Sec. 2, when , the proposal distribution degenerates to a uniform distribution over all the words in the vocabulary, and when , we recover the unigram distribution. Thus, we evaluate the impact of in the range of . Figure 4 shows the evolution of test perplexity as a function of for the RNNLMs with 256 hidden units. As can be seen, has a significant impact on the prediction accuracy. The commonly used uniform distribution (when ) and unigram distribution (when ) often yield sub-optimal solutions. For the dataset and experiment considered, gives the best perplexity (consistent on holdout set and test set). We therefore use in the experiments that follow. The number of samples used is 500, which is about 0.8% of the vocabulary size.
Figure 4 demonstrates the impact of discriminative training (6) over the maximum likelihood training (the first term of Eq. 6) on the RNNLMs with 512 hidden units using two different ’s. In general, we observe 1-3 points of perplexity reduction due to discriminative training over traditional maximum likelihood training.
Finally, we evaluate the scalability of BlackOut when number of hidden units increases. As the dataset is large, we observed that the performance of RNNLM depends on the size of the hidden layer: they perform better as the size of the hidden layer gets larger. As a truncated 64K word vocabulary is used, we interpolate the RNNLM scores with a full size 5-gram to fill in rare word probabilities (Schwenk & Gauvain, 2005; Park et al., 2010). We report the interpolated perplexities BlackOut achieved and compare them with the results from Williams et al. (2015) in Table 1. As can be seen, BlackOut reaches lower perplexities than those reported in Williams et al. (2015) within comparable time frames (often 10%-40% faster). We achieved a perplexity of 42.0 when the hidden layer size is 4096. To the best of our knowledge, this is the lowest perplexity reported on this benchmark.
|#Params||Test Perplexity||Time to Solution|
|RNN-128 + KN 5-gram||1,764||60.8||59.0||6h||9h|
|RNN-256 + KN 5-gram||1,781||57.3||55.1||16h||14h|
|RNN-512 + KN 5-gram||1,814||53.2||51.5||1d2h||1d|
|RNN-1024 + KN 5-gram||1,880||48.9||47.6||2d2h||1d14h|
|RNN-2048 + KN 5-gram||2,014||45.2||43.9||4d7h||2d15h|
|RNN-4096 + KN 5-gram||2,289||42.4||42.0||14d5h||10d|
Data from Table 1 of Williams et al. (2015).
4.2.2 When vocabulary size is 1M
In the final set of experiments, we evaluate the performance of BlackOut with a very large vocabulary of 1,000,000 words, and the results are provided in Table 2. This is the largest vocabulary used on this benchmark that we could find in existing literature. We consider the RNNLM with 1,024 hidden units (about 2 billion parameters) and 2,048 hidden units (about 4.1 billion parameters) and compare their test perplexities with the results from Le et al. (2015). We use 2,000 samples, 0.2% of the vocabulary size, for BlackOut training with . Comparing to the experiments with the 64K word vocabulary, a much smaller is used here since the sampling rate (0.2%) is much lower than that is used (0.8%) when the vocabulary size is 64K, and a smaller strikes a better balance between sample coverage per training example and convergence rate. In contrast, NCE with the same setting converges very slowly (similar to Figure 3) and couldn’t reach a competitive perplexity within the time frame considered, and its results are not reported here.
As the standard RNN/LSTM algorithms (without approximation) are used in Le et al. (2015), a cluster of 32 CPU machines (at least 20 cores each) are used to train the models for about 60 hours. BlackOut enables us to train this large model using a single CPU machine for 175 hours. Since different model architectures are used in the experiments (deep RNN/LSTM vs. standard RNNLM), the direct comparison of test perplexity isn’t very meaningful. However, this experiment demonstrates that even though our largest model is about 2-3 times larger than the models evaluated in Le et al. (2015), BlackOut, along with a few other optimization techniques, make this large scale learning problem still feasible on a single box machine without using GPUs or CPU clusters.
|Results from||LSTM (512 units)||68.8|
|Le et al. (2015)||IRNN (4 layers, 512 units)||69.4|
|60 hours||IRNN (1 layer, 1024 units + 512 linear units)||70.2|
|32 machines||RNN (4 layers, 512 tanh units)||71.8|
|RNN (1 layer, 1024 tanh units + 512 linear units)||72.5|
|Our Results||RNN (1 layer, 1024 sigmoid units)||78.4|
|175 hours, 1 machine||RNN (1 layer, 2048 sigmoid units)||68.3|
Last, we collect all the state of the art results we are aware of on this benchmark and summarize them in Table 3. Since all the models are the interpolated ones, we interpolate our best RNN model777To be consistent with the benchmark in Chelba et al. (2014), we retrained it with the full-size vocabulary of about 0.8M words. from Table 2 with the KN 5-gram model and achieve a perplexity score of 47.3. Again, different papers provide their best models trained with different architectures and vocabulary settings. Hence, an absolutely fair comparison isn’t possible. Regardless of these discrepancies, our models, within different groups of vocabulary settings, are very competitive in terms of prediction accuracy and model size.
|Model||#Params [billions]||Test Perplexity|
|RNN-1024 (full vocab) + MaxEnt||20||51.3|
|RNN-2048 (full vocab) + KN 5-gram||5.0||47.3|
|RNN-1024 (full vocab) + MaxEnt + 3 models||42.9||43.8|
|RNN-4096 (64K vocab) + KN 5-gram||2.3||42.4|
|RNN-4096 (64K vocab) + KN 5-gram||2.3||42.0|
We proposed BlackOut, a sampling-based approximation, to train RNNLMs with very large vocabularies (e.g., 1 million). We established its connections to importance sampling and noise contrastive estimation (NCE), and demonstrated its stability, sample efficiency and rate of convergence on the recently released one billion word language modeling benchmark. We achieved the lowest reported perplexity on this benchmark without using GPUs or CPU clusters.
As for future extensions, our plans include exploring other proposal distributions , and theoretical properties of the generalization property and sample complexity bounds for BlackOut. We will also investigate a multi-machine distributed implementation.
We would like to thank Oriol Vinyals, Andriy Mnih and the anonymous reviewers for their excellent comments and suggestions, which helped improve the quality of this paper.
- Andreas & Klein (2014) Andreas, Jacob and Klein, Dan. When and why are log-linear models self-normalizing? In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, 2014.
- Andreas et al. (2015) Andreas, Jacob, Rabinovich, Maxim, Klein, Dan, and Jordan, Michael I. On the accuracy of self-normalized log-linear models. In NIPS, 2015.
- Bengio & Senécal (2003) Bengio, Yoshua and Senécal, Jean-Sébastien. Quick training of probabilistic neural nets by importance sampling. In AISTATS, 2003.
- Bengio & Senécal (2008) Bengio, Yoshua and Senécal, Jean-Sébastien. Adaptive importance sampling to accelerate training of a neural probabilistic language model. In IEEE Transactions on Neural Networks, volume 19, pp. 713–722, 2008.
- Bengio et al. (2001) Bengio, Yoshua, Ducharme, Réjean, and Vincent, Pascal. A neural probabilistic language model. In NIPS, pp. 932–938, 2001.
- Bengio et al. (2013) Bengio, Yoshua, Boulanger-Lewandowski, Nicolas, and Pascanu, Razvan. Advances in optimizing recurrent networks. In ICASSP, pp. 8624–8628, 2013.
- Chelba et al. (2014) Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH, pp. 2635–2639, 2014.
- Chen et al. (2014) Chen, Xie, Wang, Yongqiang, Liu, Xunying, Gales, Mark JF, and Woodland, Philip C. Efficient gpu-based training of recurrent neural network language models using spliced sentence bunch. In INTERSPEECH, 2014.
- Chen et al. (2015) Chen, Xie, Liu, Xunying, Gales, Mark JF, and Woodland, Philip C. Recurrent neural network language model training with noise contrastive estimation for speech recognition. In ICASSP, 2015.
- Devlin et al. (2014) Devlin, Jacob, Zbib, Rabih, Huang, Zhongqiang, Lamar, Thomas, Schwartz, Richard, and Makhoul, John. Fast and robust neural network joint models for statistical machine translation. In ACL, 2014.
- Gutmann & Hyvärinen (2012) Gutmann, Michael U. and Hyvärinen, Aapo. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. JMLR, 13:307–361, 2012.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its
COURSERA: Neural Networks for Machine Learning, 2012.
Jean et al. (2015)
Jean, Sébastien, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua.
On using very large target vocabulary for neural machine translation.In ACL, 2015.
- Le et al. (2011) Le, Hai-Son, Oparin, Ilya, Allauzen, Alexandre, Gauvain, Jean-Luc, and Yvon, François. Structured output layer neural network language model. In ICASSP, pp. 5524–5527, 2011.
- Le et al. (2015) Le, Quoc V., Jaitly, Navdeep, and Hinton, Geoffrey. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arxiv:1504.00941, 2015.
- Mikolov et al. (2010) Mikolov, Tomas, Karafiát, Martin, Burget, Lukás, Cernocký, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, pp. 1045–1048, 2010.
- Mikolov et al. (2011) Mikolov, Tomas, Deoras, Anoop, Povey, Dan, Burget, Lukar, and Cernocky, Jan Honza. Strategies for training large scale neural network language models. IEEE Automatic Speech Recognition and Understanding Workshop, 2011.
- Mikolov et al. (2013) Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. Distributed representations of words and phrases and their compositionality. In Burges, Chris, Bottou, Leon, Welling, Max, Ghahramani, Zoubin, and Weinberger, Kilian (eds.), Advances in Neural Information Processing Systems 26, 2013.
- Mnih & Hinton (2008) Mnih, Andriy and Hinton, Geoffrey E. A scalable hierarchical distributed language model. In NIPS, volume 21, pp. 1081–1088, 2008.
- Mnih & Teh (2012) Mnih, Andriy and Teh, Yee Whye. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the International Conference on Machine Learning, 2012.
Morin & Bengio (2005)
Morin, Frederic and Bengio, Yoshua.
Hierarchical probabilistic neural network language model.
Proceedings of the international workshop on artificial intelligence and statistics, pp. 246–252. Citeseer, 2005.
- Park et al. (2010) Park, Junho, Liu, Xunying, Gales, Mark J. F., and Woodland, P. C. Improved neural network based language modelling and adaptation. In Proc. ISCA Interspeech, pp. 1041–1044, 2010.
- Rumelhart et al. (1988) Rumelhart, David E., Hinton, Geoffrey E., and Williams, Ronald J. Neurocomputing: Foundations of research. chapter Learning Representations by Back-propagating Errors, pp. 696–699. MIT Press, Cambridge, MA, USA, 1988.
Schwenk & Gauvain (2005)
Schwenk, Holger and Gauvain, Jean-Luc.
Training neural network language models on very large corpora.
Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 201–208, 2005.
- Shrivastava & Li (2014) Shrivastava, Anshumali and Li, Ping. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In NIPS, volume 27, pp. 2321––2329, 2014.
- Srivastava et al. (2014) Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15:1929–1958, 2014.
Sundermeyer et al. (2013)
Sundermeyer, Martin, Oparin, Ilya, Gauvain, Jean-Luc, Freiberg, Ben,
”uter, Ralf, and Ney, Hermann. Comparison of feedforward and recurrent neural network language models. In ICASSP, pp. 8430–8434, 2013.
- Vijayanarasimhan et al. (2014) Vijayanarasimhan, Sudheendra, Shlens, Jonathon, Monga, Rajat, and Yagnik, Jay. Deep networks with large output spaces. arXiv preprint arxiv:1412.7479, 2014.
- Vincent et al. (2015) Vincent, Pascal, de Brébisson, Alexandre, and Bouthillier, Xavier. Efficient exact gradient update for training deep networks with very large sparse targets. In NIPS, 2015.
- Williams et al. (2015) Williams, Will, Prasad, Niranjani, Mrva, David, Ash, Tom, and Robinson, Tony. Scaling recurrent neural network language models. In ICASSP, 2015.
Appendix A Noise distribution
The noise distribution function defined in Eq. 13 is a probability distribution function under the expectation that samples in are drawn from randomly, , such that and .
Appendix B Perplexities on Training set
Appendix C Subnet Update With Approximated RMSProp
RMSProp (Hinton, 2012) is an adaptive learning rate method that has found much success in practice. Instead of using a single learning rate to all the model parameters in , RMSProp dedicates a learning rate for each model parameter and normalizes the gradient by an exponential moving average of the magnitude of the gradient:
where denotes the decay rate. The model update at time step is then given by
where is the learning rate and is a damping factor, e.g., . While RMSProp is one of the most effective learning rate scheduling techniques, it requires a large amount of memory to store per-parameter in addition to model parameter and their gradients.
It is expensive to access and update large models with billions of parameters. Fortunately, due to the 1-of- encoding at input layer and the BlackOut sampling at output layer, the model update on and is sparse, e.g., only the model parameters corresponding to input/output words and the samples in are to be updated.888The parameter update on is still dense, but its size is several orders of magnitude smaller than those of and . For Eq. 16, however, even a model parameter is not involved in the current training, its value still needs to be updated by since its . Ignoring this update has detrimental effect on the predictive performance; in our experiments, we observed point perplexity loss if we ignore this update completely.
We resort to an approximation to . Given is the probability of a word
being selected for update, the number of time steps elapsed when it is successfully selected follows a geometric distribution with a success rate, whose mean value is . Assume that an input/output word is selected according to the unigram distribution and the samples in are drawn from , Eq. 16 can be approximated by
where is the mini-batch size and is the BPTT block size. Now we can only update the model parameters, typically a tiny fraction of , that are really involved in the current training, and thus speed up the RNNLM training further.