1 Introduction
Time series prediction algorithms have been widely applied in many areas, e.g., financial market prediction [Wu et al.2013], weather forecasting [Chakraborty et al.2012], and complex dynamical system analysis [Liu and Hauskrecht2015]. Although the wellknown autoregressive moving average (ARMA) model [Whittle1951] and its variants [Asteriou and Hall2011, Brockwell and Davis2009] have shown their effectiveness for various real world applications, they cannot model nonlinear relationships and do not differentiate among the exogenous (driving) input terms. To address this issue, various nonlinear autoregressive exogenous (NARX) models [Lin et al.1996, Gao and Er2005, Diaconescu2008, Yan et al.2013] have been developed. Typically, given the previous values of the target series, i.e. with , as well as the current and past values of driving (exogenous) series, i.e., with , the NARX model aims to learn a nonlinear mapping to the current value of target series , i.e., , where is the mapping function to learn.
Despite the fact that a substantial effort has been made for time series prediction via kernel methods [Chen et al.2008], ensemble methods [Bouchachia and Bouchachia2008], and Gaussian processes [Frigola and Rasmussen2014], the drawback is that most of these approaches employ a predefined nonlinear form and may not be able to capture the true underlying nonlinear relationship appropriately. Recurrent neural networks (RNNs) [Rumelhart et al.1986, Werbos1990, Elman1991], a type of deep neural network specially designed for sequence modeling, have received a great amount of attention due to their flexibility in capturing nonlinear relationships. In particular, RNNs have shown their success in NARX time series forecasting in recent years [Gao and Er2005, Diaconescu2008]. Traditional RNNs, however, suffer from the problem of vanishing gradients [Bengio et al.1994]
and thus have difficulty capturing longterm dependencies. Recently, long shortterm memory units (LSTM)
[Hochreiter and Schmidhuber1997]and the gated recurrent unit (GRU)
[Cho et al.2014b] have overcome this limitation and achieved great success in various applications, e.g.[Bahdanau et al.2014], speech recognition [Graves et al.2013], and image processing [Karpathy and Li2015]. Therefore, it is natural to consider stateoftheart RNN methods, e.g., encoderdecoder networks [Cho et al.2014b, Sutskever et al.2014] and attention based encoderdecoder networks [Bahdanau et al.2014], for time series prediction.Based upon LSTM or GRU units, encoderdecoder networks [Kalchbrenner and Blunsom2013, Cho et al.2014a, Cho et al.2014b, Sutskever et al.2014] have become popular due to their success in machine translation. The key idea is to encode the source sentence as a fixedlength vector and use the decoder to generate a translation. One problem with encoderdecoder networks is that their performance will deteriorate rapidly as the length of input sequence increases [Cho et al.2014a]. In time series analysis, this could be a concern since we usually expect to make predictions based upon a relatively long segment of the target series as well as driving series. To resolve this issue, the attentionbased encoderdecoder network [Bahdanau et al.2014] employs an attention mechanism to select parts of hidden states across all the time steps. Recently, a hierarchical attention network [Yang et al.2016], which uses two layers of attention mechanism to select relevant encoder hidden states across all the time steps, was also developed. Although attentionbased encoderdecoder networks and hierarchical attention networks have shown their efficacy for machine translation, image captioning [Xu et al.2015], and document classification, they may not be suitable for time series prediction. This is because when multiple driving (exogenous) series are available, the network cannot explicitly select relevant driving series to make predictions. In addition, they have mainly been used for classification, rather than time series prediction.
To address these aforementioned issues, and inspired by some theories of human attention [Hübner et al.2010]
that posit that human behavior is wellmodeled by a twostage attention mechanism, we propose a novel dualstage attentionbased recurrent neural network (DARNN) to perform time series prediction. In the first stage, we develop a new attention mechanism to adaptively extract the relevant driving series at each time step by referring to the previous encoder hidden state. In the second stage, a temporal attention mechanism is used to select relevant encoder hidden states across all time steps. These two attention models are well integrated within an LSTMbased recurrent neural network (RNN) and can be jointly trained using standard back propagation. In this way, the DARNN can adaptively select the most relevant input features as well as capture the longterm temporal dependencies of a time series appropriately. To justify the effectiveness of the DARNN, we compare it with stateoftheart approaches using the SML 2010 dataset and the NASDAQ 100 Stock dataset with a large number of driving series. Extensive experiments not only demonstrate the effectiveness of the proposed approach, but also show that the DARNN is easy to interpret, and robust to noisy inputs.
2 DualStage AttentionBased RNN
In this section, we first introduce the notation we use in this work and the problem we aim to study. Then, we present the motivation and details of the DARNN for time series prediction.
2.1 Notation and Problem Statement
Given driving series, i.e., , where is the length of window size, we use to represent a driving series of length and employ to denote a vector of exogenous (driving) input series at time .
Typically, given the previous values of the target series, i.e. with , as well as the current and past values of driving (exogenous) series, i.e., with , the NARX model aims to learn a nonlinear mapping to the current value of the target series :
(1) 
where is a nonlinear mapping function we aim to learn.
2.2 Model
Some theories of human attention [Hübner et al.2010] argue that behavioral results are best modeled by a twostage attention mechanism. The first stage selects the elementary stimulus features while the second stage uses categorical information to decode the stimulus. Inspired by these theories, we propose a novel dualstage attentionbased recurrent neural network (DARNN) for time series prediction. In the encoder, we introduce a novel input attention mechanism that can adaptively select the relevant driving series. In the decoder, a temporal attention mechanism is used to automatically select relevant encoder hidden states across all time steps. For the objective, a square loss is used. With these two attention mechanisms, the DARNN can adaptively select the most relevant input features and capture the longterm temporal dependencies of a time series. A graphical illustration of the proposed model is shown in Figure 1.
2.2.1 Encoder with input attention
The encoder is essentially an RNN that encodes the input sequences into a feature representation in machine translation [Cho et al.2014b, Sutskever et al.2014]. For time series prediction, given the input sequence with , where is the number of driving (exogenous) series, the encoder can be applied to learn a mapping from to (at time step ) with
(2) 
where is the hidden state of the encoder at time , is the size of the hidden state, and
is a nonlinear activation function that could be an LSTM
[Hochreiter and Schmidhuber1997] or gated recurrent unit (GRU) [Cho et al.2014b]. In this paper, we use an LSTM unit as to capture longterm dependencies. Each LSTM unit has a memory cell with the state at time . Access to the memory cell will be controlled by three sigmoid gates: forget gate , input gate and output gate . The update of an LSTM unit can be summarized as follows:(3) 
(4) 
(5) 
(6) 
(7) 
where is a concatenation of the previous hidden state and the current input . , , , , and , , , are parameters to learn. and
are a logistic sigmoid function and an elementwise multiplication, respectively. The key reason for using an LSTM unit is that the cell state sums activities over time, which can overcome the problem of vanishing gradients and better capture longterm dependencies of time series.
Inspired by the theory that the human attention system can select elementary stimulus features in the early stages of processing [Hübner et al.2010], we propose an input attentionbased encoder that can adaptively select the relevant driving series, which is of practical meaning in time series prediction.
Given the th input driving (exogenous) series , we can construct an input attention mechanism via a deterministic attention model, i.e.
, a multilayer perceptron, by referring to the previous hidden state
and the cell state in the encoder LSTM unit with:(8) 
and
(9) 
where , and are parameters to learn.We omit the bias terms in Eqn. (8) to be succinct. is the attention weight measuring the importance of the th input feature (driving series) at time . A softmax function is applied to to ensure all the attention weights sum to 1. The input attention mechanism is a feed forward network that can be jointly trained with other components of the RNN. With these attention weights, we can adaptively extract the driving series with
(10) 
Then the hidden state at time can be updated as:
(11) 
where is an LSTM unit that can be computed according to Eqn. (3)  (7) with replaced by the newly computed . With the proposed input attention mechanism, the encoder can selectively focus on certain driving series rather than treating all the input driving series equally.
2.2.2 Decoder with temporal attention
To predict the output , we use another LSTMbased recurrent neural network to decode the encoded input information. However, as suggested by cho2014properties [cho2014properties], the performance of the encoderdecoder network can deteriorate rapidly as the length of the input sequence increases. Therefore, following the encoder with input attention, a temporal attention mechanism is used in the decoder to adaptively select relevant encoder hidden states across all time steps. Specifically, the attention weight of each encoder hidden state at time is calculated based upon the previous decoder hidden state and the cell state of the LSTM unit with
(12) 
and
(13) 
where is a concatenation of the previous hidden state and cell state of the LSTM unit. , and are parameters to learn. The bias terms here have been omitted for clarity. The attention weight represents the importance of the th encoder hidden state for the prediction. Since each encoder hidden state is mapped to a temporal component of the input, the attention mechanism computes the context vector as a weighted sum of all the encoder hidden states ,
(14) 
Note that the context vector is distinct at each time step.
Once we get the weighted summed context vectors, we can combine them with the given target series :
(15) 
where is a concatenation of the decoder input and the computed context vector . Parameters and map the concatenation to the size the decoder input. The newly computed can be used for the update of the decoder hidden state at time :
(16) 
We choose the nonlinear function as an LSTM unit [Hochreiter and Schmidhuber1997], which has been widely used in modeling longterm dependencies. Then can be updated as:
(17) 
(18) 
(19) 
(20) 
(21) 
where is a concatenation of the previous hidden state and the decoder input . , , , , and , , , are parameters to learn. and are a logistic sigmoid function and an elementwise multiplication, respectively.
For NARX modeling, we aim to use the DARNN to approximate the function
so as to obtain an estimate of the current output
with the observation of all inputs as well as previous outputs. Specifically, can be obtained with(22) 
where is a concatenation of the decoder hidden state and the context vector. The parameters and map the concatenation to the size of the decoder hidden states. The linear function with weights and bias produces the final prediction result.
Dataset 

size  

train  valid  test  
SML 2010  16  3,200  400  537  
NASDAQ 100 Stock  81  35,100  2,730  2,730 
2.2.3 Training procedure
We use minibatch stochastic gradient descent (SGD) together with the Adam optimizer
[Kingma and Ba2014] to train the model. The size of the minibatch is 128. The learning rate starts from 0.001 and is reduced by 10% after each 10000 iterations. The proposed DARNN is smooth and differentiable, so the parameters can be learned by standard back propagation with mean squared error as the objective function:(23) 
where
is the number of training samples. We implemented the DARNN in the Tensorflow framework
[Abadi et al.2015].3 Experiments
In this section, we first describe two datasets for empirical studies. Then, we introduce the parameter settings of DARNN and the evaluation metrics. Finally, we compare the proposed DARNN against four different baseline methods, interpret the input attention as well as the temporal attention of DARNN, and study its parameter sensitivity.
3.1 Datasets and Setup
To test the performance of different methods for time series prediction, we use two different datasets as shown in Table 1.
SML 2010 is a public dataset used for indoor temperature forecasting. This dataset is collected from a monitor system mounted in a domestic house. We use the room temperature as the target series and select 16 relevant driving series that contain approximately 40 days of monitoring data. The data was sampled every minute and was smoothed with 15 minute means. In our experiment, we use the first data points as the training set, the following data points as the validation set, and the last data points as the test set.
In the NASDAQ 100 Stock dataset^{1}^{1}1http://cseweb.ucsd.edu/yaq007/NASDAQ100stockdata.html, we collected the stock prices of 81 major corporations under NASDAQ 100, which are used as the driving time series. The index value of the NASDAQ 100 is used as the target series. The frequency of the data collection is minutebyminute. This data covers the period from July 26, 2016 to December 22, 2016, 105 days in total. Each day contains 390 data points from the opening to closing of the market except that there are 210 data points on November 25 and 180 data points on December 22. In our experiments, we use the first 35,100 data points as the training set and the following 2,730 data points as the validation set. The last 2,730 data points are used as the test set. This dataset is publicly available and will be continuously enlarged to aid the research in this direction.
3.2 Parameter Settings and Evaluation Metrics
There are three parameters in the DARNN, i.e., the number of time steps in the window , the size of hidden states for the encoder , and the size of hidden states for the decoder . To determine the window size , we conducted a grid search over . The one () that achieves the best performance over validation set is used for test. For the size of hidden states for encoder () and decoder (), we set for simplicity and conduct grid search over . Those two (i.e, , ) that achieve the best performance over the validation set are used for evaluation. For all the RNN based approaches (i.e.
, NARX RNN, EncoderDecoder, Attention RNN, InputAttnRNN and DARNN), we train them 10 times and report their average performance and standard deviations for comparison.
To measure the effectiveness of various methods for time series prediction, we consider three different evaluation metrics. Among them, root mean squared error (RMSE) [Plutowski et al.1996] and mean absolute error (MAE) are two scaledependent measures, and mean absolute percentage error (MAPE) is a scaledependent measure. Specifically, assuming is the target at time and is the predicted value at time , RMSE is defined as and MAE is denoted as . When comparing the prediction performance across different datasets, mean absolute percentage error is popular because it measures the prediction deviation proportion in terms of the true values, i.e., .
3.3 ResultsI: Time Series Prediction
To demonstrate the effectiveness of the DARNN, we compare it against 4 baseline methods. Among them, the autoregressive integrated moving average (ARIMA) model is a generalization of an autoregressive moving average (ARMA) model [Asteriou and Hall2011]. NARX recurrent neural network (NARX RNN) is a classic method to address time series prediction [Diaconescu2008]. The encoderdecoder network (EncoderDecoder) [Cho et al.2014b] and attentionbased encoderdecoder network (Attention RNN) [Bahdanau et al.2014]
were originally used for machine translation tasks, in which each time step of the decoder output should be used to produce a probability distribution over the translated word codebook. To perform time series prediction, we modify these two approaches by changing the output to be a single scalar value, and use a squared loss as the objective function (as we did for the DARNN). The input to these networks is no longer words or word representations, but the
scalar driving series of length . Additionally, the decoder has the additional input of the previous values of the target series as the given information.Furthermore, we show the effectiveness of DARNN via stepbystep justification. Specifically, we compare dualstage attentionbased recurrent neural network (DARNN) against the setting that only employs its input attention mechanism (InputAttnRNN). For all RNNbased methods, the encoder takes driving series of length as the input and the decoder takes the previous values of the target series as the given information for fair comparison.
The time series prediction results of DARNN and baseline methods over the two datasets are shown in Table 2.
In Table 2, we observe that the RMSE of ARIMA is generally worse than RNN based approaches. This is because ARIMA only considers the target series and ignores the driving series . For RNN based approaches, the performance of NARX RNN and EncoderDecoder are comparable. Attention RNN generally outperforms EncoderDecoder since it is capable to select relevant hidden states across all the time steps in the encoder. Within DARNN, the input attention RNN (InputAttnRNN (128)) consistently outperforms EncoderDecoder as well as Attention RNN. This suggests that adaptively extracting driving series can provide more reliable input features to make accurate predictions. With integration of the input attention mechanism as well as temporal attention mechanism, our DARNN achieves the best MAE, MAPE, and RMSE across two datasets since it not only uses an input attention mechanism to extract relevant driving series, but also employs a temporal attention mechanism to select relevant hidden features across all time steps.
For visual comparison, we show the prediction result of EncoderDecoder (), Attention RNN () and DARNN () over the NASDAQ 100 Stock dataset in Figure 2. We observe that DARNN generally fits the ground truth much better than EncoderDecoder and Attention RNN.
3.4 ResultsII: Interpretation
To study the effectiveness of the input attention mechanism within DARNN, we test it with noisy driving (exogenous) series as the input. Specifically, within NASDAQ 100 Stock dataset, we generate 81 additional noisy driving series by randomly permuting the original 81 driving series. Then, we put these 81 noisy driving series together with the 81 original driving series as the input and test the effectiveness of DARNN. When the length of time steps is 10 and the size of hidden states is , DARNN achieves MAE: , MAPE: (0.56 0.01) and RMSE: , which are comparable to its performance in Table 2. This indicates that DARNN is robust to noisy inputs.
To further investigate the input attention mechanism, we plot the input attention weights of DARNN for the 162 input driving series (the first 81 are original and the last 81 are noisy) in Figure 3. The plotted attention weights in Figure 3 are taken from a single encoder time step and similar patterns can also be observed for other time steps. We find that the input attention mechanism can automatically assign larger weights for the 81 original driving series and smaller weights for the 81 noisy driving series in an online fashion using the activation of the input attention network to scale these weights. This demonstrates that input attention mechanism can aid DARNN to select relevant input driving series and suppress noisy input driving series.
To investigate the effectiveness of the temporal attention mechanism within DARNN, we compare DARNN to InputAttnRNN when the length of time steps varies from 3, 5, 10, 15, to 25. The detailed results over two datasets are shown in Figure 4. We observe that when is relatively large, DARNN can significantly outperform InputAttnRNN. This suggests that temporal attention mechanism can capture longterm dependencies by selecting relevant encoder hidden states across all the time steps.
3.5 ResultsIII: Parameter Sensitivity
We study the sensitivity of DARNN with respect to its parameters, i.e., the length of time steps and the size of hidden states for encoder (decoder ). When we vary or (), we keep the others fixed. By setting , we plot the RMSE versus different lengths of time steps in the window in Figure 4. It is easily observed that the performance of DARNN and InputAttnRNN will be worse when the length of time steps is too short or too long while DARNN is relatively more robust than InputAttnRNN. By setting , we also plot the RMSE versus different sizes of hidden states for encoder and decoder () in Figure 5. We notice that DARNN usually achieves the best performance when or . Moreover, we can also conclude that DARNN is more robust to parameters than InputAttnRNN.
4 Conclusion
In this paper, we proposed a novel dualstage attentionbased recurrent neural network (DARNN), which consists of an encoder with an input attention mechanism and a decoder with a temporal attention mechanism. The newly introduced input attention mechanism can adaptively select the relevant driving series. The temporal attention mechanism can naturally capture the longrange temporal information of the encoded inputs. Based upon these two attention mechanisms, the DARNN can not only adaptively select the most relevant input features, but can also capture the longterm temporal dependencies of a time series appropriately. Extensive experiments on the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrated that our proposed DARNN can outperform stateoftheart methods for time series prediction.
The proposed dualstage attentionbased recurrent neural network (DARNN) not only can be used for time series prediction, but also has the potential to serve as a general feature learning tool in computer vision tasks
[Pu et al.2016, Qin et al.2015]. In the future, we are going to employ DARNN to perform ranking and binary coding [Song et al.2015, Song et al.2016].Acknowledgments
GWC is supported in part by NSF cooperative agreement SMA 1041755 to the Temporal Dynamics of Learning Center, and a gift from Hewlett Packard. GWC and YQ were also partially supported by Guangzhou Science and Technology Planning Project (Grant No. 201704030051).
References

[Abadi et al.2015]
Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen,
Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al.
Tensorflow: Largescale machine learning on heterogeneous systems.
2015.  [Asteriou and Hall2011] Dimitros Asteriou and Stephen G Hall. Arima models and the boxjenkins methodology. Applied Econometrics, 2(2):265–286, 2011.
 [Bahdanau et al.2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.
 [Bengio et al.1994] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning longterm dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994.
 [Bouchachia and Bouchachia2008] Abdelhamid Bouchachia and Saliha Bouchachia. Ensemble learning for time series prediction. In Proceedings of the 1st International Workshop on Nonlinear Dynamics and Synchronization, 2008.
 [Brockwell and Davis2009] Peter J. Brockwell and Richard A Davis. Time Series: Theory and Methods (2nd ed.). Springer, 2009.
 [Chakraborty et al.2012] Prithwish Chakraborty, Manish Marwah, Martin F Arlitt, and Naren Ramakrishnan. Finegrained photovoltaic output prediction using a bayesian ensemble. In AAAI, 2012.
 [Chen et al.2008] S. Chen, X. X. Wang, and C. J. Harris. Narxbased nonlinear system identification using orthogonal least squares basis hunting. IEEE Transactions on Control Systems Technology, 16(1):78–84, 2008.
 [Cho et al.2014a] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoderdecoder approaches. arXiv:1409.1259, 2014.
 [Cho et al.2014b] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv:1406.1078, 2014.
 [Diaconescu2008] Eugen Diaconescu. The use of NARX neural networks to predict chaotic time series. WSEA Transactions on Computer Research, 3(3), 2008.
 [Elman1991] Jeffrey L Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(23):195–225, 1991.
 [Frigola and Rasmussen2014] R. Frigola and C. E. Rasmussen. Integrated preprocessing for bayesian nonlinear system identification with gaussian processes. In IEEE Conference on Decision and Control, pages 552–560, 2014.
 [Gao and Er2005] Yang Gao and Meng Joo Er. Narmax time series model prediction: feedforward and recurrent fuzzy neural network approaches. Fuzzy Sets and Systems, 150(2):331–350, 2005.
 [Graves et al.2013] Alex Graves, Abdelrahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, pages 6645–6649, 2013.
 [Hochreiter and Schmidhuber1997] Sepp Hochreiter and Jürgen Schmidhuber. Long shortterm memory. Neural Computation, 9(8):1735–1780, 1997.
 [Hübner et al.2010] Ronald Hübner, Marco Steinhauser, and Carola Lehle. A dualstage twophase model of selective attention. Psychological Review, 117(3):759–784, 2010.
 [Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In EMNLP, volume 3, pages 413–422, 2013.
 [Karpathy and Li2015] Andrej Karpathy and FeiFei Li. Deep visualsemantic alignments for generating image descriptions. In CVPR, pages 3128–3137, 2015.
 [Kingma and Ba2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
 [Lin et al.1996] Tsungnan Lin, Bill G. Horne, Peter Tino, and C. Lee Giles. Learning longterm dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks, 7(6):1329–1338, 1996.
 [Liu and Hauskrecht2015] Zitao Liu and Milos Hauskrecht. A regularized linear dynamical system framework for multivariate time series analysis. In AAAI, pages 1798–1805, 2015.
 [Plutowski et al.1996] Mark Plutowski, Garrison Cottrell, and Halbert White. Experience with selecting exemplars from clean data. Neural Networks, 9(2):273–294, 1996.
 [Pu et al.2016] Yunchen Pu, Martin Renqiang Min, Zhe Gan, and Lawrence Carin. Adaptive feature abstraction for translating video to language. arXiv preprint arXiv:1611.07837, 2016.
 [Qin et al.2015] Yao Qin, Huchuan Lu, Yiqun Xu, and He Wang. Saliency detection via cellular automata. In CVPR, pages 110–119, 2015.
 [Rumelhart et al.1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by backpropagating errors. Nature, 323(9):533–536, 1986.
 [Song et al.2015] Dongjin Song, Wei Liu, Rongrong Ji, David A Meyer, and John R Smith. Top rank supervised binary coding for visual search. In ICCV, pages 1922–1930, 2015.
 [Song et al.2016] Dongjin Song, Wei Liu, and David A Meyer. Fast structural binary coding. In IJCAI, pages 2018–2024, 2016.
 [Sutskever et al.2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112, 2014.
 [Werbos1990] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990.
 [Whittle1951] P. Whittle. Hypothesis Testing in Time Series Analysis. PhD thesis, 1951.
 [Wu et al.2013] Yue Wu, José Miguel HernándezLobato, and Zoubin Ghahramani. Dynamic covariance models for multivariate financial time series. In ICML, pages 558–566, 2013.
 [Xu et al.2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, volume 14, pages 77–81, 2015.
 [Yan et al.2013] Linjun Yan, Ahmed Elgamal, and Garrison W. Cottrell. Substructure vibration NARX neural network approach for statistical damage inference. Journal of Engineering Mechanics, 139:737–747, 2013.
 [Yang et al.2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In NAACL, 2016.