# Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networks

Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.

## Authors

• 2 publications
• 1 publication
• 2 publications
• ### Joint Learning of Correlated Sequence Labelling Tasks Using Bidirectional Recurrent Neural Networks

The stream of words produced by Automatic Speech Recognition (ASR) syste...
03/14/2017 ∙ by Vardaan Pahuja, et al. ∙ 0

• ### Training Neural Networks with Stochastic Hessian-Free Optimization

Hessian-free (HF) optimization has been successfully used for training d...
01/16/2013 ∙ by Ryan Kiros, et al. ∙ 0

• ### GeoSeq2Seq: Information Geometric Sequence-to-Sequence Networks

The Fisher information metric is an important foundation of information ...
10/25/2017 ∙ by Alessandro Bay, et al. ∙ 0

• ### Deep Recurrent Convolutional Neural Network: Improving Performance For Speech Recognition

A deep learning approach has been widely applied in sequence modeling pr...
11/22/2016 ∙ by Zewang Zhang, et al. ∙ 0

• ### Quaternion Recurrent Neural Networks

Recurrent neural networks (RNNs) are powerful architectures to model seq...
06/12/2018 ∙ by Titouan Parcollet, et al. ∙ 0

• ### Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form

In this paper we focus on the linear algebra theory behind feedforward (...
09/16/2017 ∙ by Maxim Naumov, et al. ∙ 0

• ### CITlab ARGUS for Arabic Handwriting

In the recent years it turned out that multidimensional recurrent neural...
12/15/2014 ∙ by Gundram Leifert, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Multidimensional recurrent neural networks (MDRNNs) constitute an efficient architecture for building a multidimensional context into recurrent neural networks [1]. End-to-end training of MDRNNs in conjunction with connectionist temporal classification (CTC) has been shown to achieve a state-of-the-art performance in on/off-line handwriting and speech recognition [2, 3, 4].

In previous approaches, the performance of MDRNNs having a depth of up to five layers, which is limited as compared to the recent progress in feed¬forward networks [5], was demonstrated. The effectiveness of MDRNNs deeper than five layers has thus far been unknown.

Training a deep architecture has always been a challenging topic in machine learning. A notable breakthrough was achieved when deep feedforward neural networks were initialized using layer-wise pre-training

[6]. Recently, approaches have been proposed in which supervision is added to intermediate layers to train deep networks [5, 7]. To the best of our knowledge, no such pre-training or bootstrapping method has been developed for MDRNNs.

Alternatively, Hesssian-free (HF) optimization is an appealing approach to training deep neural networks because of its ability to overcome pathological curvature of the objective function [8]. Furthermore, it can be applied to any connectionist model provided that its objective function is differentiable. The recent success of HF for deep feedforward and recurrent neural networks [8, 9] supports its application to MDRNNs.

In this paper, we claim that an MDRNN can benefit from a deeper architecture, and the application of second order optimization such as HF allows its successful learning. First, we offer details of the development of HF optimization for MDRNNs. Then, to apply HF optimization for sequence labeling tasks, we address the problem of the non-convexity of CTC, and formulate a convex approximation. In addition, its relationship with the EM algorithm and the Fisher information matrix is discussed. Experimental results for offline handwriting and phoneme recognition show that an MDRNN with HF optimization performs better as the depth of the network increases up to 15 layers.

## 2 Multidimensional recurrent neural networks

MDRNNs constitute a generalization of RNNs to process multidimensional data by replacing the single recurrent connection with as many connections as the dimensions of the data [1]. The network can access the contextual information from

directions, allowing a collective decision to be made based on rich context information. To enhance its ability to exploit context information, long short-term memory (LSTM)

[10] cells are usually utilized as hidden units. In addition, stacking MDRNNs to construct deeper networks further improves the performance as the depth increases, achieving the state-of-the-art performance in phoneme recognition [4]

. For sequence labeling, CTC is applied as a loss function of the MDRNN. The important advantage of using CTC is that no pre-segmented sequences are required, and the entire transcription of the input sample is sufficient.

### 2.1 Learning MDRNNs

A -dimensional MDRNN with inputs and outputs is regarded as a mapping from an input sequence to an output sequence of length , where the input data for

input neurons are given by the vectorization of

-dimensional data, and is the length of the sequence in each dimension. All learnable weights and biases are concatenated to obtain a parameter vector . In the learning phase with fixed training data, the MDRNN is formalized as a mapping from the parameters to the output sequence a, i.e., . The scalar loss function is defined over the output sequence as . Learning an MDRNN is viewed as an optimization of the objective with respect to .

The Jacobian of a function is the matrix where each element is a partial derivative of an element of output with respect to an element of input. The Hessian of a scalar function is the matrix of second-order partial derivatives of the output with respect to its inputs. Throughout this paper, a vector sequence is denoted by boldface a, a vector at time in a is denoted by , and the -th element of is denoted by .

## 3 Hessian-free optimization for MDRNNs

The application of HF optimization to an MDRNN is straightforward if the matching loss function [11] for its output layer is adopted. However, this is not the case for CTC, which is necessarily adopted for sequence labeling. Before developing an appropriate approximation to CTC that is compatible with HF optimization, we discuss two considerations related to the approximation. The first is obtaining a quadratic approximation of the loss function, and the second is the efficient calculation of the matrix-vector product used at each iteration of the conjugate gradient (CG) method.

HF optimization minimizes an objective by constructing a local quadratic approximation for the objective function and minimizing the approximate function instead of the original one. The loss function needs to be approximated at each point of the -th iteration:

 Qn(θ)=L(θn)+∇θL|⊤θnδn+12δ⊤nGδn, (1)

where is the search direction, i.e., the parameters of the optimization, and is a local approximation to the curvature of at , which is typically obtained by the generalized Gauss-Newton (GGN) matrix as an approximation of the Hessian.

HF optimization uses the CG method in a subroutine to minimize the quadratic objective above for utilizing the complete curvature information and achieving computational efficiency. CG requires the computation of for an arbitrary vector , but not the explicit evaluation of . For neural networks, an efficient way to compute was proposed in [11], extending the study in [12]. In section 3.2, we provide the details of the efficient computation of for MDRNNs.

### 3.1 Quadratic approximation of loss function

The Hessian matrix, , of the objective is written as

 HL∘N=J⊤NHLJN+KT∑i=1[JL]iH[N]i, (2)

where , , and denotes the -th component of the vector . An indefinite Hessian matrix is problematic for second-order optimization, because it defines an unbounded local quadratic approximation [13]. For nonlinear systems, the Hessian is not necessarily positive semidefinite, and thus, the GGN matrix is used as an approximation of the Hessian [11, 8]. The GGN matrix is obtained by ignoring the second term in Eq. (2), as given by

 GL∘N=J⊤NHLJN. (3)

The sufficient condition for the GGN approximation to be exact is that the network makes a perfect prediction for every given sample, that is, , or stays in the linear region for all , that is, .

has less rank than and is positive semidefinite provided that is. Thus, is chosen to be a convex function so that is positive semidefinite. In principle, it is best to define and such that performs as much of the computation as possible, with the positive semidefiniteness of as a minimum requirement [13]. In practice, a nonlinear output layer together with its matching loss function [11], such as the softmax function with cross-entropy loss, is widely used.

### 3.2 Computation of matrix-vector product for MDRNN

The product of an arbitrary vector by the GGN matrix, , amounts to the sequential multiplication of by three matrices. First, the product is a Jacobian times vector and is therefore equal to the directional derivative of along the direction of . Thus, can be written using a differential operator [12] and the properties of the operator can be utilized for efficient computation. Because an MDRNN is a composition of differentiable components, the computation of

throughout the whole network can be accomplished by repeatedly applying the sum, product, and chain rules starting from the input layer. The detailed derivation of the

operator to LSTM, normally used as a hidden unit in MDRNNs, is provided in appendix A.

Next, the multiplication of by can be performed by direct computation. The dimension of could at first appear problematic, since the dimension of the output vector used by the loss function can be as high as , in particular, if CTC is adopted as an objective for the MDRNN. If the loss function can be expressed as the sum of individual loss functions with a domain restricted in time, the computation can be reduced significantly. For example, with the commonly used cross-entropy loss function, the matrix can be transformed into a block diagonal matrix with blocks of a Hessian matrix. Let be the -th block in . Then, the GGN matrix can be written as

 GL∘N=∑tJ⊤NtHL,tJNt, (4)

where is the Jacobian of the network at time .

Finally, the multiplication of a vector by the matrix

is calculated using the backpropagation through time algorithm by propagating

instead of the error at the output layer.

## 4 Convex approximation of CTC for application to HF optimization

Connectioninst temporal classification (CTC) [14] provides an objective function of learning an MDRNN for sequence labeling. In this section, we derive a convex approximation of CTC inspired by the GGN approximation according to the following steps. First, the non-convex part of the original objective is separated out by reformulating the softmax part. Next, the remaining convex part is approximated without altering its Hessian, making it well matched to the non-convex part. Finally, the convex approximation is obtained by reuniting the convex and non-convex parts.

### 4.1 Connectionist temporal classification

CTC is formulated as the mapping from an output sequence of the recurrent network, , to a scalar loss. The output activations at time are normalized using the softmax function

 ytk=exp(atk)∑k′exp(atk′), (5)

where

is the probability of label

given a at time .

The conditional probability of the path is calculated by the multiplication of the label probabilities at each timestep, as given by

 p(π|a)=T∏t=1ytπt, (6)

where is the label observed at time along the path . The path of length is mapped to a label sequence of length by an operator , which removes the repeated labels and then the blanks. Several mutually exclusive paths can map to the same label sequence. Let be a set containing every possible sequence mapped by , that is, is the image of , and let denote the cardinality of the set.

The conditional probability of a label sequence l is given by

 p(l|a)=∑π∈B−1(l)p(π|a), (7)

which is the sum of probabilities of all the paths mapped to a label sequence l by .

The cross-entropy loss assigns a negative log probability to the correct answer. Given a target sequence z, the loss function of CTC for the sample is written as

 L(a)=−logp(z|a). (8)

From the description above, CTC is composed of the sum of the product of softmax components. The function , corresponding to the softmax with cross-entropy loss, is convex [11]. Therefore, is log-concave. Whereas log-concavity is closed under multiplication, the sum of log-concave functions is not log-concave in general [15]. As a result, the CTC objective is not convex in general because it contains the sum of softmax components in Eq. (7).

### 4.2 Reformulation of CTC objective function

We reformulate the CTC objective Eq. (8) to separate out the terms that are responsible for the non-convexity of the function. By reformulation, the softmax function is defined over the categorical label sequences.

By substituting Eq. (5) into Eq. (6), it follows that

 p(π|a)=exp(bπ)∑π′∈allexp(bπ′), (9)

where . By substituting Eq. (9) into Eq. (7) and setting , can be re-written as

 p(z|a)=∑π∈B−1(z)exp(bπ)∑π∈allexp(bπ)=exp(fz)∑z′∈Sexp(fz′), (10)

where is the set of every possible label sequence and is the log-sum-exp function111 is the log-sum-exp function defined on , which is proportional to the probability of observing the label sequence z among all the other label sequences.

With the reformulation above, the CTC objective can be regarded as the cross-entropy loss with the softmax output, which is defined over all the possible label sequences. Because the cross-entropy loss function matches the softmax output layer [11], the CTC objective is convex, except the part that computes for each of the label sequences. At this point, an obvious candidate for the convex approximation of CTC is the GGN matrix separating the convex and non-convex parts.

Let the non-convex part be and the convex part be . The mapping is defined by

 Nc(a)=F=[fz1,…,fz|S|]⊤, (11)

where is given above, and is the number of all the possible label sequences. For given as above, the mapping is defined by

 Lc(F)=−logexp(fz)∑%z′∈Sexp(fz′)=−fz+log⎛⎜⎝∑z′∈Sexp(fz′)⎞⎟⎠, (12)

where z is the label sequence corresponding to a. The final reformulation for the loss function of CTC is given by

 L(a)=Lc∘Nc(a). (13)

### 4.3 Convex approximation of CTC loss function

The GGN approximation of Eq. (13) immediately gives a convex approximation of the Hessian for CTC as . Although has the form of a diagonal matrix plus a rank- matrix, i.e., , the dimension of is , where becomes exponentially large as the length of the sequence increases. This makes the practical calculation of difficult.

On the other hand, removing the linear team from in Eq. (12) does not alter its Hessian. The resulting formula is . The GGN matrices of and are the same, i.e., . Therefore, their Hessian matrices are approximations of each other. The condition that the two Hessian matrices, and , converges to the same matrix is discussed below.

Interestingly, is given as a compact formula , where is the output unit at time . Its Hessian can be directly computed, resulting in a block diagonal matrix. Each block is restricted in time, and the -th block is given by

 HM,t=diag(Yt)−YtYt⊤, (14)

where and is given in Eq. (5). Because the Hessian of each block is positive semidefinite, is positive semidefinite. A convex approximation of the Hessian of an MDRNN using the CTC objective can be obtained by substituting for in Eq. (3). Note that the resulting matrix is block diagonal and Eq. (4) can be utilized for efficient computation.

Our derivation can be summarized as follows:

1. is not positive semidefinite.

2. is positive semidefinite, but not computationally tractable.

3. is positive semidefinite and computationally tractable.

### 4.4 Sufficient condition for the proposed approximation to be exact

From Eq. (2), the condition holds if and only if . Since in general, we consider only the case of for all , which corresponds to the case where is a linear mapping.

contains a log-sum-exp function mapping from paths to a label sequence. Let l be the label sequence corresponding to ; then, for . If the probability of one path is sufficiently large to ignore all the other paths, that is, for , it follows that . This is a linear mapping, which results in .

In conclusion, the condition holds if one dominant path exists such that for each label sequence l.

### 4.5 Derivation of the proposed approximation from the Fisher information matrix

The identity of the GGN and the Fisher information matrix [16] has been shown for the network using the softmax with cross-entropy loss [17, 18]. Thus, it follows that the GGN matrix of Eq. (13) is identical to the Fisher information matrix. Now, we show that the proposed matrix in Eq. (14) is derived from the Fisher information matrix under the condition given in section 4.4. The Fisher information matrix of an MDRNN using CTC is written as

 F=Ex⎡⎣J⊤NEl∼p(l|a)⎡⎣(∂logp(l|a)∂a)⊤(∂logp(l|a)∂a)⎤⎦JN⎤⎦, (15)

where is the -dimensional output of the network . CTC assumes output probabilities at each timestep to be independent of those at other timesteps [1], and therefore, its Fisher information matrix is given as the sum of every timestep. It follows that

 F=Ex⎡⎣∑tJ⊤NtEl∼p(l|a)⎡⎣(∂logp(l|a)∂at)⊤(∂logp(l|a)∂at)⎤⎦JNt⎤⎦. (16)

Under the condition in section 4.4, the Fisher information matrix is given by

 F=Ex[∑tJ⊤Nt(diag(Yt)−YtYt⊤)JNt], (17)

which is the same form as Eqs. (4) and (14) combined. See appendix B for the detailed derivation.

### 4.6 EM interpretation of the proposed approximation

The goal of the Expectation-Maximization (EM) algorithm is to find the maximum likelihood solution for models having latent variables

[19]. Given an input sequence x, and its corresponding target label sequence z, the log likelihood of z is given by , where represents the model parameters. For each observation x, we have a corresponding latent variable which is a 1-of- binary vector where is the number of all the paths mapped to z. The log likelihood can be written in terms of as . The EM algorithm starts with an initial parameter , and repeats the following process until convergence.

Expectation step calculates: .

Maximization step updates: , where .

In the context of CTC and RNN, is given as as in Eq. (6), where is the -dimensional output of the neural network. Taking the second-order derivative of with respect to gives , with as in Eq. (14). Because this term is independent of and , the Hessian of with respect to is given by

 HQ,t=diag(Yt)−YtYt⊤, (18)

which is the same as the convex approximation in Eq. (14).

## 5 Experiments

In this section, we present the experimental results for two different sequence labeling tasks, offline handwriting recognition and phoneme recognition. The performance of Hessian-free optimization for MDRNNs with the proposed matrix is compared with that of stochastic gradient descent (SGD) optimization on the same settings.

### 5.1 Database and preprocessing

The IFN/ENIT Database [20] is a database of handwritten Arabic words, which consists of 32,492 images. The entire dataset has five subsets (a, b, c, d, e). The 25,955 images corresponding to the subsets were used for training. The validation set consisted of 3,269 images corresponding to the first half of the sorted list in alphabetical order (ae07_001.tif ai54_028.tif) in set a. The remaining images in set a

, amounting to 3,268, were used for the test. The intensity of pixels was centered and scaled using the mean and standard deviation calculated from the training set.

The TIMIT corpus [21] is a benchmark database for evaluating speech recognition performance. The standard training, validation, and core datasets were used. Each set contains 3,696 sentences, 400 sentences, and 192 sentences, respectively. A mel spectrum with 26 coefficients was used as a feature vector with a pre-emphasis filter, 25 ms window size, and 10 ms shift size. Each input feature was centered and scaled using the mean and standard deviation of the training set.

### 5.2 Experimental setup

For handwriting recognition, the basic architecture was adopted from that proposed in [3]. Deeper networks were constructed by replacing the top layer with more layers. The number of LSTM cells in the augmented layer was chosen such that the total number of weights between the different networks was similar. The detailed architectures are described in Table 1, together with the results.

For phoneme recognition, the deep bidirectional LSTM and CTC in [4] was adopted as the basic architecture. In addition, the memory cell block [10], in which the cells share the gates, was applied for efficient information sharing. Each LSTM block was constrained to have 10 memory cells.

According to the results, using a large value of bias for input/output gates is beneficial for training deep MDRNNs. A possible explanation is that the activation of neurons is exponentially decayed by input/output gates during the propagation. Thus, setting large bias values for these gates may facilitate the transmission of information through many layers at the beginning of the learning. For this reason, the biases of the input and output gates were initialized to 2, whereas those of the forget gates and memory cells were initialized to 0. All the other weight parameters of the MDRNN were initialized randomly from a uniform distribution in the range

.

The label error rate was used as the metric for performance evaluation, together with the average loss of CTC in Eq. (8). It is defined by the edit distance, which sums the total number of insertions, deletions, and substitutions required to match two given sequences. The final performance, shown in Tables  1 and 2, was evaluated using the weight parameters that gave the best label error rate on the validation set. To map output probabilities to a label sequence, best path decoding [1] was used for handwriting recognition and beam search decoding [4, 22] with a beam width of 100 was used for phoneme recognition. For phoneme recognition, 61 phoneme labels were used during training and decoding, and then, mapped to 39 classes for calculating the phoneme error rate (PER) [4, 23].

For phoneme recognition, the regularization method suggested in [24] was used. We applied Gaussian weight noise of standard deviation together with L2 regularization of strength . The network was first trained without noise, and then, it was initialized to the weights that gave the lowest CTC loss on the validation set. Then, the network was retrained with Gaussian weight noise [4]. Table 2 presents the best result for different values of .

#### 5.2.1 Parameters

For HF optimization, we followed the basic setup described in [8]

, but different parameters were utilized. Tikhonov damping was used together with Levenberg-Marquardt heuristics. The value of the damping parameter

was initialized to 0.1, and adjusted according to the reduction ratio (multiplied by 0.9 if , divided by 0.9 if , and unchanged otherwise). The initial search direction for each run of CG was set to the CG direction found by the previous HF optimization iteration decayed by 0.7. To ensure that CG followed the descent direction, we continued to perform a minimum and maximum of additional CG iterations after it found the first descent direction. We terminated CG at iteration before reaching the maximum iteration if the following condition was satisfied: , where

is the quadratic objective of CG without offset. The training data were divided into 100 and 50 mini-batches for the handwriting and phoneme recognition experiments, respectively, and used for both the gradient and matrix-vector product calculation. The learning was stopped if any of two criteria did not improve for 20 epochs and 10 epochs in handwriting and phoneme recognition, respectively.

For SGD optimization, the learning rate was chosen from , and the momentum from . For handwriting recognition, the best performance obtained using all the possible combinations of parameters is presented in Table 1. For phoneme recognition, the best parameters out of nine candidates for each network were selected after training without weight noise based on the CTC loss. Additionally, the backpropagated error in LSTM layer was clipped to remain in the range for stable learning [25]. The learning was stopped after 1000 epochs had been processed, and the final performance was evaluated using the weight parameters that showed the best label error rate on the validation set. It should be noted that in order to guarantee the convergence, we selected a conservative criterion as compared to the study where the network converged after 85 epochs in handwriting recognition [3] and after 55-150 epochs in phoneme recognition [4].

### 5.3 Results

Table 1 presents the label error rate on the test set for handwriting recognition. In all cases, the networks trained using HF optimization outperformed those using SGD. The advantage of using HF is more pronounced as the depth increases. The improvements resulting from the deeper architecture can be seen with the error rate dropping from 6.1% to 4.5% as the depth increases from 3 to 13.

Table 2 shows the phoneme error rate (PER) on the core set for phoneme recognition. The improved performance according to the depth can be observed for both optimization methods. The best PER for HF optimization is 18.54% at 15 layers and that for SGD is 18.46% at 10 layers, which are comparable to that reported in [4], where the reported results are a PER of 18.6% from a network with 3 layers having 3.8 million weights and a PER of 18.4% from a network with 5 layers having 6.8 million weights. The benefit of a deeper network is obvious in terms of the number of weight parameters, although this is not intended to be a definitive performance comparison because of the different preprocessing. The advantage of HF optimization is not prominent in the result of the experiments using the TIMIT database. One explanation is that the networks tend to overfit to a relatively small number of the training data samples, which removes the advantage of using advanced optimization techniques.

## 6 Conclusion

Hessian-free optimization as an approach for successful learning of deep MDRNNs, in conjunction with CTC, was presented. To apply HF optimization to CTC, a convex approximation of its objective func¬tion was explored. In experiments, improvements in performance were seen as the depth of the network increased for both HF and SGD. HF optimization showed a significantly better performance for handwriting recognition than did SGD, and a comparable performance for speech recognition.

## References

• Graves [2012a] Alex Graves. Supervised sequence labelling with recurrent neural networks, volume 385. Springer, 2012a.
• Graves et al. [2008] Alex Graves, Marcus Liwicki, Horst Bunke, Jürgen Schmidhuber, and Santiago Fernández. Unconstrained on-line handwriting recognition with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 577–584, 2008.
• Graves and Schmidhuber [2009] Alex Graves and Jürgen Schmidhuber. Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in Neural Information Processing Systems, pages 545–552, 2009.
• Graves et al. [2013] Alex Graves, Abdel-ranhman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP, pages 6645–6649. IEEE, 2013.
• Romero et al. [2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/1412.6550, 2014. URL http://arxiv.org/abs/1412.6550.
• Hinton and Salakhutdinov [2006] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.
• Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

, pages 1–9, 2015.
• Martens [2010] James Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning, pages 735–742, 2010.
• Martens and Sutskever [2011] James Martens and Ilya Sutskever. Learning recurrent neural networks with Hessian-free optimization. In Proceedings of the 28th International Conference on Machine Learning, pages 1033–1040, 2011.
• Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
• Schraudolph [2002] Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7):1723–1738, 2002.
• Pearlmutter [1994] Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural Computation, 6(1):147–160, 1994.
• Martens and Sutskever [2012] James Martens and Ilya Sutskever. Training deep and recurrent networks with Hessian-free optimization. In Neural Networks: Tricks of the Trade, pages 479–535. Springer, 2012.
• Graves et al. [2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, pages 369–376, 2006.
• Boyd and Vandenberghe [2004] Stephen Boyd and Lieven Vandenberghe, editors. Convex Optimization. Cambridge University Press, 2004.
• Amari [1998] Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276, 1998.
• Pascanu and Bengio [2014] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In International Conference on Learning Representations, 2014.
• Park et al. [2000] Hyeyoung Park, S-I Amari, and Kenji Fukumizu. Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13(7):755–764, 2000.
• Bishop [2007] Christopher M. Bishop, editor. Pattern Recognition and Machine Learning. Springer, 2007.
• Pechwitz et al. [2002] Mario Pechwitz, S Snoussi Maddouri, Volker Märgner, Noureddine Ellouze, and Hamid Amiri. IFN/ENIT-database of handwritten arabic words. In Proceedings of CIFED, pages 129–136, 2002.
• DARPA-ISTO [1990] DARPA-ISTO. The DARPA TIMIT acoustic-phonetic continuous speech corpus (TIMIT). In speech disc cd1-1.1 edition, 1990.
• Graves [2012b] Alex Graves. Sequence transduction with recurrent neural networks. In ICML Representation Learning Workshop, 2012b.
• Lee and Hon [1989] Kai-Fu Lee and Hsiao-Wuen Hon.

Speaker-independent phone recognition using hidden markov models.

IEEE Transactions on Acoustics, Speech and Signal Processing, 37(11):1641–1648, 1989.
• Graves [2011] Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348–2356, 2011.
• Graves [2008] Alex Graves. Rnnlib: A recurrent neural network library for sequence learning problems, 2008.

## Appendix A Derivation of the R operator to LSTM

We follow the version of LSTM in [4]. The forward pass of LSTM is given by

 it =σ(Wxixt+Whiht−1+Wcict−1+bi), ft =σ(Wxfxt+Whfht−1+Wcfct−1+bf), ct =ft⋅ct−1+it⋅tanh(Wxcxt+Whcht−1+bc), ot =σ(Wxoxt+Whoht−1Wcoct<