1 Introduction
Sequence generation and prediction tasks span many modes of data, ranging from audio and language modelling, to more general timeseries prediction tasks. Applications of such models include speech recognition, machine translation, dialogue generation, speech synthesis, forecasting, and music generation, among others. Neural networks can be applied to these tasks by predicting sequence elements onebyone, conditioning on the history of sequence elements, forming an autoregressive model. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs), including longshort term memory (LSTM) networks
(Hochreiter & Schmidhuber, 1997) in particular, have achieved many successes at these tasks. However, in their basic form, these models have a limited ability to adapt to recently observed parts of a sequence.Many sequences contain repetition; a pattern that occurs once is more likely to occur again. For instance, a word that occurs once in a document is much more likely to occur again. A sequence of handwriting will generally stay in the same handwriting style. A sequence of speech will generally stay in the same voice. Although RNNs have a hidden state that can summarize the recent past, they are often unable to exploit new patterns that occur repeatedly in a test sequence.
This paper concerns dynamic evaluation, which we investigate as a candidate solution to this problem. Our approach adapts models to recent sequences using gradient descent based mechanisms. We show several ways to improve on past dynamic evaluation approaches in Section 5, and use our improved methodology to achieve stateoftheart results in Section 7.1 and Section 7.2. In Section 6 we design a method to dramatically to reduce the number of adaptation parameters in dynamic evaluation, making it practical in a wider range of situations. In Section 7.3 we analyse dynamic evaluation’s performance over varying timescales and distribution shifts, and demonstrate that dynamically evaluated models can generate conditional samples that repeat many patterns from the conditioning data.
2 Motivation
Generative models can assign probabilities to sequences by modelling each term in the factorization given by the product rule. The probability of a sequence factorizes as
(1) 
Methods that apply this factorization either use a fixed context when predicting
, for instance as in Ngrams or CNNs, or use a recurrent hidden state to summarize the context, as in an RNN. However, for longer sequences, the history
often contains reoccurring patterns that are difficult to capture using models with fixed parameters (static models).In many domains, in a dataset of sequences , each sequence is generated from a slightly different distribution . At any point in time , the history of a sequence contains useful information about the generating distribution for that specific sequence . Therefore adapting the model parameters learned during training is justified. We aim to infer a set of model parameters from that will better approximate within sequence .
Many sequence modelling tasks are characterised by sequences generated from slightly different distributions as in the scenario described above. The generating distribution may also change continuously across a single sequence; for instance, a text excerpt may change topic. Furthermore, many machine learning benchmarks do not distinguish between sequence boundaries, and concatenate all sequences into one continuous sequence. Thus, many sequence modelling tasks could be seen as having a local distribution
as well as a global distribution . During training time, the goal is to find the best fixed model possible for . However, during evaluation time, a model that can infer the current from the recent history has an advantage.3 Dynamic evaluation
Dynamic evaluation methods continuously adapt the model parameters , learned at training time, to parts of a sequence during evaluation. The goal is to learn adapted parameters that provide a better model of the local sequence distribution, . When dynamic evaluation is applied in the present work, a long test sequence is divided up into shorter sequences of length . We define to be a sequence of shorter sequence segments
(2) 
The initial adapted parameters are set to , and used to compute the probability of the first segment, . This probability gives a cross entropy loss , with gradient , which is computed using truncated backpropagation through time (Werbos, 1990). The gradient is used to update the model, resulting in adapted parameters , before evaluating . The same procedure is then repeated for , and for each in the sequence as shown in Figure 1. Gradients for each loss
are only backpropagated to the beginning of
, so computation is linear in the sequence length. Each update applies one maximum likelihood training step to approximate the current local distribution . The computational cost of dynamic evaluation is one forward pass and one gradient computation through the data, with some slight overhead to apply the update rule for every sequence segment.As in all autoregressive models, dynamic evaluation only conditions on sequence elements that it has already predicted, and so evaluates a valid logprobability for each sequence. Dynamic evaluation can also be used while generating sequences. In this case, the model generates each sequence segment using fixed weights, and performs a gradient descent based update step on . Applying dynamic evaluation for sequence generation could result in generated sequences with more consistent regularities, meaning that patterns that occur in the generated sequence are more likely to occur again.
4 Background
4.1 Related approaches
Adaptive language modelling was first considered for ngrams, adapting to recent history via caching (Jelinek et al., 1991; Kuhn, 1988), and other methods Bellegarda (2004). More recently, the neural cache approach (Grave et al., 2017) and the closely related pointer sentinelLSTM (Merity et al., 2017b) have been used to for adaptive neural language modelling. Neural caching has recently been used to improve the stateoftheart at wordlevel language modelling (Merity et al., 2017a).
The neural cache model learns a type of nonparametric output layer on the fly at test time, which allows the network to adapt to recent observations. Each past hidden state is paired with the next input , and is stored as a tuple . When a new hidden state is observed, the output probabilities are adjusted to give a higher weight to output words that coincided with past hidden states with a large inner product .
(3) 
where
is a one hot encoding of
, andis a scaling parameter. The cache probabilities are interpolated with the base network probabilities to adapt the base network at test time.
The neural cache closely relates to dynamic evaluation, as both methods can be added on top of a base model for adaptation at test time. The main difference is the mechanism used to fit to recent history: the neural cache approach uses a nonparametric, nearest neighbourslike method, whereas dynamic evaluation uses a gradient descent based method to change model parameters dynamically. Both methods rely on an autoregressive factorisation, as they depend on observing sequence elements after they are predicted in order to perform adaptation. Dynamic evaluation and neural caching methods are therefore both applicable to sequence prediction and generation tasks, but not directly to more general supervised learning tasks.
One drawback of the neural cache method is that it cannot adjust the recurrent hidden state dynamics. As a result, the neural cache’s ability to capture information that occurs jointly between successive sequence elements is limited. This capability is critical for adapting to sequences where each element has very little independent meaning, e.g. character level language modelling.
4.2 Dynamic evaluation in neural networks
Dynamic evaluation of neural language models was proposed by Mikolov et al. (2010)
. Their approach simply used stochastic gradient descent (SGD) updates at every time step, computing the gradient with fully truncated backpropagation through time, which is equivalent to setting
in equation (2). Dynamic evaluation has since been applied to character and wordlevel language models (Graves, 2013; Krause et al., 2017; Ororbia II et al., 2017; Fortunato et al., 2017). Previous work using dynamic evaluation considered it as an aside, and did not explore it in depth.5 Update Rule methodology for Dynamic evaluation
We propose several changes to Mikolov et al. (2010)’s dynamic evaluation method with SGD and fully truncated backpropagation, which we refer to as traditional dynamic evaluation. The first modification reduces the update frequency, so that gradients are backpropagated over more timesteps. This change provides more accurate gradient information, and also improves the computational efficiency of dynamic evaluation, since the update rule is applied much less often. We use sequence segments of length 5 for wordlevel tasks and 20 for characterlevel tasks.
Next, we add a global decay prior to bias the model towards the parameters learned during training. Our motivation for dynamic evaluation assumes that the local generating distribution is constantly changing, so it is potentially desirable to weight recent sequence history higher in adaptation. Adding a global decay prior accomplishes this by causing previous adaptation updates to decay exponentially over time. For SGD with a global prior, learning rate and decay rate ; we form the update rule
(4) 
We then consider using an RMSprop
(Tieleman & Hinton, 2012) derived update rule for the learning rule in place of SGD. RMSprop uses a moving average of recent squared gradients to scale learning rates for each weight. In dynamic evaluation, near the start of a test sequence, RMSprop has had very few gradients to average, and therefore may not be able to leverage its updates as effectively. For this reason, we collect mean squared gradients, , on the training data rather than on recent test data (which is what RMSprop would do). is given by(5) 
where is the number of training batches and is the gradient on the th training batch. The minibatch size for this computation becomes a hyperparameter, as larger minibatches will result in smaller mean squared gradients. The update rule, which we call RMS with a global prior in our experiments, is then
(6) 
where is a stabilization parameter. For the decay step of our update rule, we also consider scaling the decay rate for each parameter proportionally to . Parameters with a high RMS gradient affect the dynamics of the network more, so it makes sense to decay them faster. is divided by its mean, resulting in a normalized version of with a mean of 1:
(7) 
We clip the values of to be no greater than to be sure that the decay rate does not exceed for any parameter. Combining the learning component and the regularization component results in the final update equation, which we refer to as RMS with an RMS global prior
(8) 
6 Sparse dynamic evaluation
Minibatching over sequences is desirable for some testtime sequence modelling applications because it allows faster processing of multiple sequences in parallel. Dynamic evaluation has a high memory cost for minibatching because it is necessary to store a different set of parameters for each sequence in the minibatch. Therefore, we consider a sparse dynamic evaluation variant that updates a smaller number of parameters. We introduce a new adaptation matrix which is initialized to zeros.
multiplies hidden state vector
of an RNN at every timestep to get a new hidden state , via(9) 
then replaces and is propagated throughout the network via both recurrent and feedforward connections. Applying dynamic evaluation to avoids the need to apply dynamic evaluation to the original parameters of the network, reduces the number of adaptation parameters, and makes minibatching less memory intensive. We reduce the number of adaptation parameters further by only using to transform an arbitrary subset of hidden units. This results in being an matrix with adaptation parameters. If is chosen to be much less than the number of hidden units, this reduces the number of adaptation parameters dramatically. In Section 7.2 we experiment with sparse dynamic evaluation for characterlevel language models.
7 Experiments
We applied dynamic evaluation to wordlevel and characterlevel language modelling. In all tasks, we evaluate dynamic evaluation on top of a base model. After training the base model, we tune hyperparameters for dynamic evaluation on the validation set, and evaluate both the static and dynamic versions of the model on the test set. We also consider follow up experiments that analyse the sequence lengths for which dynamic evaluation is useful. Code for our dynamic evaluation methodology is available^{1}^{1}1https://github.com/benkrause/dynamicevaluation.
7.1 Wordlevel language modelling
We train base models on the Penn Treebank (PTB, Marcus et al., 1993) and WikiText2 (Merity et al., 2017b) datasets, and compare the performance of static and dynamic evaluation. These experiments compare dynamic evaluation against past approaches such as the neural cache and measure dynamic evaluation’s general performance across different models and datasets.
PTB is derived from articles of the Wall Street Journal. It contains 929k training tokens and a vocab size limited to 10k words. It is one of the most commonly used benchmarks in language modelling. We consider two baseline models on PTB, a standard LSTM implementation with recurrent dropout (Zaremba et al., 2014), and the recent stateoftheart AWDLSTM (Merity et al., 2017a).
Our standard LSTM was taken from the Chainer tutorial on language modelling^{2}^{2}2https://github.com/chainer/chainer/tree/master/examples/ptb, and used two LSTM layers with 650 units each, trained with SGD and regularized with recurrent dropout. On our standard LSTM, we experiment with traditional dynamic evaluation as applied by Mikolov et al. (2010), as well as each modification we make building up to our final update rule as described in Section 5. As our final update rule (RMS + RMS global prior) worked best, we use this for all other experiments and use “dynamic eval” by default to refer to this update rule in tables.
We applied dynamic evaluation on a more powerful model, the ASGD weightdropped LSTM (AWDLSTM, Merity et al., 2017a). The AWDLSTM is a vanilla LSTM that combines the use of dropconnect (Wan et al., 2013) on recurrent weights for regularization, and a variant of averaged stochastic gradient descent (Polyak & Juditsky, 1992) for optimisation. Our model, which used 3 layers and tied input and output embeddings (Press & Wolf, 2017; Inan et al., 2017), was intended to be a direct replication of AWDLSTM, using code from their implementation^{3}^{3}3https://github.com/salesforce/awdlstmlm. Results are given in Table 1.
model  parameters  valid  test 
RNN+LDA+kN5+cache (Mikolov & Zweig, 2012)  92.0  
CharCNN (Kim et al., 2016)  19M  78.9  
LSTM (Zaremba et al., 2014)  66M  82.2  78.4 
Variational LSTM (Gal & Ghahramani, 2016)  66M  73.4  
Pointer sentinelLSTM (Merity et al., 2017b)  21M  72.4  70.9 
Variational LSTM + augmented loss (Inan et al., 2017)  51M  71.1  68.5 
Variational RHN (Zilly et al., 2017)  23M  67.9  65.4 
NAS cell (Zoph & Le, 2017)  54M  62.4  
Variational LSTM + gradual learning (Aharoni et al., 2017)  105M  61.7  
LSTM + BB tuning (Melis et al., 2017)  24M  60.9  58.3 
LSTM (Grave et al., 2017)  86.9  82.3  
LSTM + neural cache (Grave et al., 2017)  74.6  72.1  
LSTM (ours)  20M  88.0  85.6 
LSTM + traditional dynamic eval (sgd, bptt=1)  20M  78.6  76.2 
LSTM + dynamic eval (sgd, bptt=5)  20M  78.0  75.6 
LSTM + dynamic eval (sgd, bptt=5, global prior)  20M  77.4  74.8 
LSTM + dynamic eval (RMS, bptt=5, global prior)  20M  74.3  72.2 
LSTM + dynamic eval (RMS, bptt=5, RMS global prior)  20M  73.5  71.7 
AWDLSTM (Merity et al., 2017a)  24M  60.0  57.3 
AWDLSTM +neural cache (Merity et al., 2017a)  24M  53.9  52.8 
AWDLSTM (ours)  24M  59.8  57.7 
AWDLSTM + dynamic eval  24M  51.6  51.1 
Dynamic evaluation gives significant overall improvements to both models on this dataset. Dynamic evaluation also achieves better final results than the neural cache on both a standard LSTM and the AWDLSTM reimplementation, and improves the stateoftheart on PTB.
WikiText2 is roughly twice the size of PTB, with 2 million training tokens and a vocab size of 33k. It features articles in a nonshuffled order, with dependencies across articles that adaptive methods should be able to exploit. For this dataset, we use the same baseline LSTM implementation and AWDLSTM reimplementation as on PTB. Results are given in Table 2.
model  parameters  valid  test 

Byte mLSTM (Krause et al., 2016)  46M  92.8  88.8 
Variational LSTM (Inan et al., 2017)  28M  91.5  87.0 
Pointer sentinelLSTM (Merity et al., 2017b)  84.8  80.8  
LSTM + BB tuning (Melis et al., 2017)  24M  69.1  65.9 
LSTM (Grave et al., 2017)  104.2  99.3  
LSTM + neural cache (Grave et al., 2017)  72.1  68.9  
LSTM (ours)  50M  109.1  103.4 
LSTM + dynamic eval  50M  63.7  59.8 
AWDLSTM (Merity et al., 2017a)  33M  68.6  65.8 
AWDLSTM + neural cache (Merity et al., 2017a)  33M  53.8  52.0 
AWDLSTM (ours)  33M  68.9  66.1 
AWDLSTM + dynamic eval  33M  46.4  44.3 
Dynamic evaluation improves the stateoftheart perplexity on WikiText2, and provides a significantly greater improvement than neural caching to both base models. This suggests that dynamic evaluation is effective at exploiting regularities that cooccur across nonshuffled documents.
7.2 Characterlevel language modelling
We consider dynamic evaluation on the text8^{4}^{4}4http://mattmahoney.net/dc/textdata, and Hutter Prize (Hutter, 2006) datasets. The Hutter Prize dataset is comprised of Wikipedia text, and includes XML and characters from nonLatin languages. It is 100 million UTF8 bytes long and contains 205 unique bytes. Similarly to other reported results, we use a 9055 split for training, validation, and testing. The text8 dataset is also derived from Wikipedia text, but has all XML removed, and is lower cased to only have 26 characters of English text plus spaces. As with Hutter Prize, we use the standard 9055 split for training, validation, and testing for text8. We used a multiplicative LSTM (mLSTM) (Krause et al., 2016)^{5}^{5}5https://github.com/benkrause/mLSTM as our base model for both datasets. The mLSTMs for both tasks used 2800 hidden units, an embedding layer of 400 units, weight normalization (Salimans & Kingma, 2016), variational dropout (Gal & Ghahramani, 2016), and ADAM (Kingma & Ba, 2014) for training.
We also consider sparse dynamic evaluation, as described in Section 6, on the Hutter Prize dataset. For sparse dynamic evaluation, we adapted a subset of hidden units, resulting in a adaptation matrix and 250k adaptation parameters. All of our dynamic evaluation results in this section use the final update rule given in Section 5. Results for Hutter Prize are given in Table 3, and results for text8 are given in Table 4.
model  parameters  test 
Stacked LSTM (Graves, 2013)  21M  1.67 
Stacked LSTM + traditional dynamic eval (Graves, 2013)  21M  1.33 
Multiplicative integration LSTM (Wu et al., 2016)  17M  1.44 
HyperLSTM (Ha et al., 2017)  27M  1.34 
Hierarchical multiscale LSTM (Chung et al., 2017)  1.32  
Bytenet decoder (Kalchbrenner et al., 2016)  1.31  
LSTM + BB tuning (Melis et al., 2017)  46M  1.30 
Recurrent highway networks (Zilly et al., 2017)  46M  1.27 
Fastslow LSTM (Mujika et al., 2017)  47M  1.25 
mLSTM (Krause et al., 2016)  46M  1.24 
mLSTM + sparse dynamic eval ()  46M  1.13 
mLSTM + dynamic eval  46M  1.08 
model  parameters  test 
Multiplicative RNN (Mikolov et al., 2012)  5M  1.54 
Multiplicative integration LSTM (Wu et al., 2016)  4M  1.44 
LSTM (Cooijmans et al., 2017)  1.43  
Batch normalised LSTM (Cooijmans et al., 2017)  1.36  
Hierarchical multiscale LSTM (Chung et al., 2017)  1.29  
Recurrent highway networks (Zilly et al., 2017)  45M  1.27 
mLSTM (Krause et al., 2016)  45M  1.27 
mLSTM + dynamic eval  45M  1.19 
Dynamic evaluation achieves large improvements to our base models and stateoftheart results on both datasets. Sparse dynamic evaluation also achieves significant improvements on Hutter Prize using only 0.5% of the adaptation parameters of regular dynamic evaluation.
7.3 Timescales of dynamic evaluation
We measure timescales at which dynamic evaluation gains an advantage over static evaluation. Starting from the model trained on Hutter Prize, we plot the performance of static and dynamic evaluation against the number of characters processed on sequences from the Hutter Prize test set, and sequences in Spanish from the European Parliament dataset (Koehn, 2005).
The Hutter Prize data experiments show the timescales at which dynamic evaluation gained the advantage observed in Table 3. We divided the Hutter Prize test set into 500 sequences of length 10000, and applied static and dynamic evaluation to these sequences using the same model and methodology used to obtain results in Table 3. Losses were averaged across these 500 sequences to obtain average losses at each time step. Plots of the average crossentropy errors against the number of Hutter characters sequenced are given in Figure 1(a).
The Spanish experiments measure how dynamic evaluation handles large distribution shifts between training and test time, as Hutter Prize contains very little Spanish. We used the first 5 million characters of the Spanish European Parliament data in place of the Hutter Prize test set. The Spanish experiments used the same base model and dynamic evaluation settings as Hutter Prize. Plots of the average crossentropy errors against the number of Spanish characters sequenced are given in Figure 1(b).
On both datasets, dynamic evaluation gave a very noticeable advantage after a few hundred characters. For Spanish this advantage continued to grow as more of the sequence was processed, whereas for Hutter, this advantage was maximized after viewing around 23k characters. The advantage of dynamic evaluation was also much greater on Spanish sequences than Hutter sequences.
We also drew 300 character conditional samples from the static and dynamic versions of our model after viewing 10k characters of Spanish. For the dynamic model, we continued to apply dynamic evaluation during sampling as well, by the process described in Section 3. The conditional samples are given in the appendix. The static samples quickly switched to English that resembled Hutter Prize data. The dynamic model generated data with some Spanish words and a number of made up words with characteristics of Spanish words for the entirety of the sample. This is an example of the kinds of features that dynamic evaluation was able to learn to model on the fly.
8 Conclusion
This work explores and develops methodology for applying dynamic evaluation to sequence modelling tasks. Experiments show that the proposed dynamic evaluation methodology gives large test time improvements across character and word level language modelling. Our improvements to language modelling have applications to speech recognition and machine translation over longer contexts, including broadcast speech recognition and paragraph level machine translation. Overall, dynamic evaluation is shown to be an effective method for exploiting pattern reoccurrence in sequences.
References
 Aharoni et al. (2017) Z. Aharoni, G. Rattner, and H. Permuter. Gradual learning of deep recurrent neural networks. arXiv preprint arXiv:1708.08863, 2017.
 Bellegarda (2004) J. R. Bellegarda. Statistical language model adaptation: review and perspectives. Speech Communication, 42(1):93–108, 2004.
 Chung et al. (2017) J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks. ICLR, 2017.

Cooijmans et al. (2017)
T. Cooijmans, N. Ballas, C. Laurent, and A. Courville.
Recurrent batch normalization.
ICLR, 2017.  Fortunato et al. (2017) M. Fortunato, C. Blundell, and O. Vinyals. Bayesian recurrent neural networks. arXiv preprint arXiv:1704.02798, 2017.
 Gal & Ghahramani (2016) Y. Gal and Z. Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pp. 1019–1027, 2016.
 Grave et al. (2017) E. Grave, A. Joulin, and N. Usunier. Improving neural language models with a continuous cache. ICLR, 2017.
 Graves (2013) A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
 Ha et al. (2017) D. Ha, A. Dai, and Q. Lee. Hypernetworks. ICLR, 2017.
 Hochreiter & Schmidhuber (1997) S. Hochreiter and J. Schmidhuber. Long shortterm memory. Neural Computation, 9:1735–1780, 1997.
 Hutter (2006) M. Hutter. The human knowledge compression prize. URL http://prize.hutter1.net, 2006.

Inan et al. (2017)
H. Inan, K. Khosravi, and R. Socher.
Tying word vectors and word classifiers: A loss framework for language modeling.
ICLR, 2017.  Jelinek et al. (1991) F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. A dynamic language model for speech recognition. In HLT, volume 91, pp. 293–295, 1991.
 Kalchbrenner et al. (2016) N. Kalchbrenner, L. Espeholt, K. Simonyan, A. Oord, A. Graves, and K. Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099, 2016.

Kim et al. (2016)
Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush.
Characteraware neural language models.
In
Thirtieth AAAI Conference on Artificial Intelligence
, 2016.  Kingma & Ba (2014) D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
 Koehn (2005) P. Koehn. Europarl: A parallel corpus for statistical machine translation. In MT Summit, volume 5, pp. 79–86, 2005.
 Krause et al. (2016) B. Krause, L. Lu, I. Murray, and S. Renals. Multiplicative LSTM for sequence modelling. arXiv preprint arXiv:1609.07959, 2016.
 Krause et al. (2017) B. Krause, I. Murray, S. Renals, and L. Lu. Multiplicative LSTM for sequence modelling. ICLR Workshop track, 2017. URL https://openreview.net/forum?id=SJCS5rXFl.

Kuhn (1988)
R. Kuhn.
Speech recognition and the frequency of recently used words: A modified Markov model for natural language.
In Proceedings of the 12th conference on Computational linguisticsVolume 1, pp. 348–350. Association for Computational Linguistics, 1988.  Marcus et al. (1993) M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313–330, 1993.
 Melis et al. (2017) G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589, 2017.
 Merity et al. (2017a) S. Merity, N. S. Keskar, and R. Socher. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182, 2017a.
 Merity et al. (2017b) S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. ICLR, 2017b.
 Mikolov & Zweig (2012) T. Mikolov and G. Zweig. Context dependent recurrent neural network language model. SLT, 12:234–239, 2012.
 Mikolov et al. (2010) T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010.
 Mikolov et al. (2012) T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky. Subword language modeling with neural networks. preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf), 2012.
 Mujika et al. (2017) A. Mujika, F. Meier, and A. Steger. Fastslow recurrent neural networks. arXiv preprint arXiv:1705.08639, 2017.
 Ororbia II et al. (2017) A. G. Ororbia II, T. Mikolov, and D. Reitter. Learning simpler language models with the differential state framework. Neural Computation, 2017.
 Polyak & Juditsky (1992) B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992.
 Press & Wolf (2017) O. Press and L. Wolf. Using the output embedding to improve language models. EACL 2017, pp. 157, 2017.
 Salimans & Kingma (2016) T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901–909, 2016.
 Tieleman & Hinton (2012) T. Tieleman and G. E. Hinton. Lecture 6.5rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4(2), 2012.
 Wan et al. (2013) L. Wan, M. Zeiler, S. Zhang, Yann L. Cun, and R. Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML13), pp. 1058–1066, 2013.
 Werbos (1990) P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78:1550–1560, 1990.
 Wu et al. (2016) Y. Wu, S. Zhang, Y. Zhang, Y. Bengio, and R. Salakhutdinov. On multiplicative integration with recurrent neural networks. In NIPS, 2016.
 Zaremba et al. (2014) W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
 Zilly et al. (2017) J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent highway networks. ICLR, 2017.

Zoph & Le (2017)
B. Zoph and Quoc V Le.
Neural architecture search with reinforcement learning.
ICLR, 2017.
Appendix A Appendix
a.1 Dynamic samples conditioned on Spanish
300 character samples generated from the dynamic version of the model trained on Hutter Prize, conditioned on 10k of Spanish characters. The final sentence fragment of the 10k conditioning characters is given to the reader, with the generated text given in
bold:Tiene importancia este compromiso en la medida en que la Comisión es un organismo que tiene el montembre tas procedíns la conscriptione se ha Tesalo del Pómienda que et hanemos que Pe la Siemina.
De la Pedrera Orden es Señora Presidente civil, Orden de siemin presente relevante frónmida que esculdad pludiore e formidad President de la Presidenta Antidorne Adamirmidad i ciemano de el 200’. Fo
a.2 Static samples conditioned on Spanish
300 character samples generated from the static version of the model trained on Hutter Prize, conditioned on 10k of Spanish characters. The final sentence fragment of the 10k conditioning characters is given to the reader, with the generated text given in bold:
Tiene importancia este compromiso en la medida en que la Comisión es un organismo que tiene el monde,
<br>There is a secret act in the world except Cape Town, seen in now flat comalo and ball market and has seen the closure of the eagle as imprints in a dallas within the country." Is a topic for an increasingly small contract saying Allan Roth acquired the government in [[1916]].
===