# Recurrent Orthogonal Networks and Long-Memory Tasks

Although RNNs have been shown to be powerful tools for processing sequential data, finding architectures or optimization strategies that allow them to model very long term dependencies is still an active area of research. In this work, we carefully analyze two synthetic datasets originally outlined in (Hochreiter and Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store information over many time steps. We explicitly construct RNN solutions to these problems, and using these constructions, illuminate both the problems themselves and the way in which RNNs store different types of information in their hidden states. These constructions furthermore explain the success of recent methods that specify unitary initializations or constraints on the transition matrices.

## Authors

• 11 publications
• 56 publications
• 83 publications
• ### Gated Orthogonal Recurrent Units: On Learning to Forget

We present a novel recurrent neural network (RNN) based model that combi...
06/08/2017 ∙ by Li Jing, et al. ∙ 0

• ### Depth Enables Long-Term Memory for Recurrent Neural Networks

A key attribute that drives the unprecedented success of modern Recurren...
03/23/2020 ∙ by Alon Ziv, et al. ∙ 0

• ### Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies

Recurrent neural networks (RNNs) have achieved state-of-the-art performa...
02/24/2017 ∙ by Robert DiPietro, et al. ∙ 0

• ### Twin Networks: Using the Future as a Regularizer

Being able to model long-term dependencies in sequential data, such as t...
08/22/2017 ∙ by Dmitriy Serdyuk, et al. ∙ 0

• ### Learning Longer-term Dependencies via Grouped Distributor Unit

Learning long-term dependencies still remains difficult for recurrent ne...
04/29/2019 ∙ by Wei Luo, et al. ∙ 0

• ### Population-based Global Optimisation Methods for Learning Long-term Dependencies with RNNs

Despite recent innovations in network architectures and loss functions, ...
05/23/2019 ∙ by Bryan Lim, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Recurrent Neural Networks (RNNs) are powerful models which are naturally suited to processing sequential data. They can maintain a hidden state which encodes information about previous elements in the sequence. For a classical version of RNN (Elman, 1990), at every timestep, the hidden state is updated as a function of both the input and the current hidden state. In theory, this recursive procedure allows these models to store complex signals for arbitrarily long timescales.

However, in practice RNNs are considered difficult to train due to the so-called vanishing and exploding gradient problems

(Bengio et al., 1994)

. These problems arise when the spectral norm of the transition matrix is significantly different than 1, or due to the non-linear transition functions. If the spectral norm of the transition matrix is greater than 1, the gradients will grow exponentially in magnitude during backpropagation, which is known as the exploding gradient problem. If the spectral norm is less than 1, the gradients will vanish exponentially quickly, which is known as the vanishing gradient problem. Recently, a simple strategy of clipping gradients has been introduced, and has proved effective in addressing the exploding gradient problem

(Mikolov, 2012)

. The problem of vanishing gradients has shown itself to be more difficult, and various strategies have been proposed over the years to address it. One very successful approach, known as Long Short-Term Memory (LSTM) units

(Hochreiter & Schmidhuber, 1997), has been to modify the architecture of the hidden units by introducing gates which explicitly control the flow of information as a function of both the state and the input. Specifically, the signal stored in a hidden unit must be explicitly erased by a forget gate and is otherwise stored indefinitely. This allows information to be carried over long periods of time. LSTM’s have become very successful in applications to language modeling, machine translation, and speech recognition (Wojciech Zaremba, 2014; Sutskever et al., 2014; Graves et al., 2013). Other methods have been proposed to deal with learning long-term dependencies, such as adding a separate contextual memory (Tomas Mikolov, 2015), stabilizing activations (Krueger & Mimesevic, 2015) or using more sophisticated optimization schemes (Martens & Sutskever, 2011). Two recent methods propose to directly address the vanishing gradient problem by either initializing or parameterizing the transition matrix with orthogonal or unitary matrices (Martin Arjovsky, 2015; Quoc V. Le, 2015).

These works have used a set of synthetic problems (originally outlined in (Hochreiter & Schmidhuber, 1997) or variants thereof) for testing the ability of methods to learn long-term dependencies. These synthetic problems are designed to be pathologically difficult, and require models to store information over very long timescales (hundreds of timesteps). Different approaches have solved these problems to varying degrees of success. In (Martens & Sutskever, 2011), the authors report that their Hessian-Free optimization based method solves the addition task for timesteps. The authors of  (Krueger & Mimesevic, 2015) reported that their method beat the chance baseline for the adding task in 8/9 cases for . In (Quoc V. Le, 2015), the IRNN is reported to solve the addition task for . The method proposed in (Martin Arjovsky, 2015) is able to solve the copy task for up to timesteps, and is able to completely solve the addition task for up to timesteps, and partially solves it for .

In this work we analyze these “long-memory” tasks, and construct explicit RNN solutions for them. The solutions illuminate both the tasks, and provide a theoretical justification for the success of recent approaches using orthogonal initializations of, or unitary constraints on, the transition matrix of the RNN. In particular, we show that a classical Elman RNN with no transition non-linearity and random orthogonal initialization is with high probability close to our explicit RNN solution to the sequence memorization task, and the same network architecture with identity initialization is close to the explicit solution to the addition task. We verify experimentally that initializing correctly (i.e. random orthogonal or identity) is critical for success on these tasks. Finally, we show how

pooling can be used to allow a model to “choose” between an random-orthogonal or identity-like memory.

## 2 Architectures

We review some recurrent neural network (RNN) architectures for processing sequential data, and discuss the modifications we use for the long memory problems. We fix the following notation: input sequences are denoted

, and output sequences are denoted by

### 2.1 sRNN

An sRNN (Elman, 1990) consists of a transition matrix , an decoder matrix where is the output dimension, a encoder matrix , and a bias . If either the output or input is categorical, (respectively ) is the number of classes, and we use a one-hot representation. As the sRNN ingests a sequence, it keeps running updates to a hidden state , and using the hidden state and the decoder matrix, produces outputs :

 ht=σ(Uxt+Vht−1+b)yt=Wht (1)

where are the input, output and hidden state respectively at time . While there have been great improvements in the training of sRNN’s since their introduction, and while they have shown to be powerful models in tasks such as language modeling (Mikolov, 2012), it can still be difficult to train generic sRNN’s to use information about inputs from hundreds of timesteps previous for computing the current output (Bengio et al., 1994; Pascanu et al., 2013).

In the following, we will use a simplification of the sRNN that makes them in some sense less powerful models, but makes it easier to train them to solve simple long-memory tasks. Namely, by placing the non-linearity between the input and hidden state, rather than between the hidden state and output, we obtain RNNs with linear transitions (or, in the case of categorical inputs, not using a non-linearity at all). We call these LT-RNNs. The update equations are then:

 ht=σ(Uxt+b)+Vht−1yt=Wht (2)

Finally, note that by appropriately scaling the weights and biases, a sRNN can be made to approximate a LT-RNN, but not the other way around (and of course the optimization may never find this scaling)

### 2.2 Lstm

The LSTM of (Hochreiter & Schmidhuber, 1997) is an architecture designed to improve upon the sRRN with the introduction of simple memory cells with a gating architecture. In this work we use the architecture in (Wojciech Zaremba, 2014), which is the same as that originally proposed in (Hochreiter & Schmidhuber, 1997). For each memory cell, the network computes the output of four gates: an update gate, input gate, forget gate and output gate. The outputs of these gates are:

 i=σ(Uixt+Viht−1)f=σ(Ufxt+Vfht−1)o=σ(Uoxt+Voht−1)g=tanh(Ugxt+Vght−1) (3)

The cell state is then updated as a function of the input and the previous state:

 ct=f⊙ct−1+i⊙g (4)

Finally, the hidden state is computed as a function of the cell state and the output gate:

 ht=o⊙tanh(ct) (5)

A relatively common variation of the original LSTM involves adding so-called “peephole connections” (Gers et al., 2003) which allows information to flow from the cell state to the various gates. The update equations of the gates become:

 i=σ(Uixt+Viht−1+Wict−1)f=σ(Ufxt+Vfht−1+Wfct−1)o=σ(Uoxt+Voht−1+Woct−1)g=tanh(Ugxt+Vght−1+Wgct−1) (6)

This variant was originally designed to measure or generate precise time intervals, and has proven successful for speech recognition and sequence generation (Graves, 2013; Graves et al., 2013).

### 2.3 LT-RNN with l2 pooling

Below we will consider LT-RNN’s initialized with either random orthogonal transition matrices, or identity transitions, and we will see that there is a large difference in behavior between these initializations. However, we can set up an architecture where a random orthogonal initialization behaves much closer to an identity initialization by using an pooling layer at the output. If we feed both the pooled and unpooled hidden layer to the decoder, the model can choose whether it wants an identity-like or random-orthogonal like representation. We fix a pool size , and then the update equations for this model are:

 ht=σ(Uxt+b)+Vht−1yt=WIht+WPPk(ht) (7)

where if is the

dimensional vector

, then is the dimensional vector defined by

 P(h)i= ⎷ki∑j=k(i−1)+1h2j

In this section we describe tasks from (Hochreiter & Schmidhuber, 1997; Martin Arjovsky, 2015; Quoc V. Le, 2015) which involve dependencies over very long timescales which are designed to be pathologically hard for the sRNN.

### 3.1 Copying Problem

This task tests the network’s ability to recall information seen many time steps previously. We follow the same setup as (Martin Arjovsky, 2015), which we briefly outline here. Let be a set of symbols, and pick numbers and . The input consists of a length vector of categories, starting with entries sampled uniformly from which are the sequence to be remembered. The next inputs are set to , which is a blank category. The following (single) input is , which represents a delimiter indicating that the network should output the initial entries of the input. The last inputs are set to . The required output sequence consists of entries of , followed by the first entries of the input sequence in exactly the same order. The task is to minimize the average cross-entropy of the predictions at each time step, which amounts to remembering a categorical sequence of length for time steps.

#### 3.1.1 a solution mechanism

We can write out an LT-RNN solution for this problem. We will write out descriptions for from equation (2) Fix a number . For each in , pick a random integer drawn uniformly from , and let

 Qj=(cos(2ljπ/(T+S))sin(2ljπ/(T+S))−sin(2ljπ/(T+S))cos(2ljπ/(T+S))).

Now define , and then from (2) by

 Q=⎛⎜ ⎜ ⎜ ⎜ ⎜⎝Q10⋯00Q2⋯0⋮⋱000Qd⎞⎟ ⎟ ⎟ ⎟ ⎟⎠,V=(Q001).

So is a block diagonal matrix. Note that iterating “spins” each of the at different rates, but they all synchronize at multiples of . Thus acts as a “clock” with period . Now set to be a matrix with rows sampled uniformly from the unit sphere, and form by appending two zero rows to and then one extra column, with for each entry between 1 and , for the entry, and for the entry. Schematically,

 U=⎡⎢⎣~U1/(2S+1)0−10S+T+1⎤⎥⎦.

Finally, set , except scale the row by , and zero out the row.

Now we will show how the RNN operates, for which we need a little more notation. Denote by the first coordinates of , and by the last coordinate, and denote the th row of . Then the RNN works as follows, initialized with hidden state :

• After the first inputs, we have

 ~h=S∑j=1QS−juij.
• For the next T inputs, only changes, incrementing by at each step. Note that is now the best match to because it has large negative last component.

• At time When the token is seen, is set positive.

• At time ,

 ~h=ui1+S∑j=2Q1−juij.

We argue below that if is large enough w.r.t. and , with high probability, is small, and so multiplication with has max value at .

• The sum continues to cycle, giving as output for each following up to .

We now briefly argue that is small when is large enough w.r.t. and

. We will repeatedly use that the variance of a sum of independent, mean-zero random variables grows as the sum of the variances. Denote by

to be the pair of coordinates of the th row of corresponding th block; since are uniform on the sphere, we expect

 ||uji||2∼1/d.

Since for each fixed , over the choices of in the definition of , the are independent, has mean zero, and since

 uTjQpuj=d∑i=1uTjiQpiuji

we expect

 |uTjQpuj|2∼1/d.

Moreover, since the are uniform on the sphere,

 |uTjQpuj′|2∼1/d

for . Similarly, we expect

 ∣∣ ∣∣uTi1S∑j=2Q1−juij∣∣ ∣∣2∼S/d.

Thus we can fix a small number , say , and choose large enough so that with high probability , even though . Finally, there is a weak dependence on here; for fixed and it is exponentially unlikely (in ) that the nearest neighbor is close enough to to interfere.

This solution mechanism suggests that a random orthogonal matrix (chosen, for example, via QR decomposition of a Gaussian matrix) is a good starting point for solving this task. The construction above is invariant to rotations; and we can always find a basis so that a given orthogonal matrix has the block form above in that basis. Thus all that is necessary is for the descent to nudge the eigenvalues of the orthogonal matrix to be

roots of unity, and then it already has the basic form of the construction above. This also gives a good explanation for the performance of the models used for the copy problem in (Martin Arjovsky, 2015)

Finally, note that although we used the setup of (Martin Arjovsky, 2015), the construction can be modified to solve problems 2a and 2b in (Hochreiter & Schmidhuber, 1997)

#### 3.1.2 solution mechanism experiments

Since the construction of the copy mechanism is randomized, we provide an experiment to show how the solution degrades as a function of (the dictionary size) and (the length of the sequence to be remembered). There is not a strong dependence on (the length of time to remember the sequence). Figure 1 shows the number of successes over runs with

#### 3.1.3 Variable Length Copy Problem

Note that the solution mechanism for the copy problem above depends on having a fixed location for regurgitating the input. In the experiments below, we also discuss a variant of the copy task, where the symbol to indicate that the memorized sequence must be output is randomly located in ; this can be considered a variant of task 2c in (Hochreiter & Schmidhuber, 1997). We do not know an explicit LT-RNN or sRNN solution for this variable length problem (although the above solution using a multiplicative RNN instead of an sRNN, and keeping extra hidden variables to track the power of solves it).

The adding problem requires the network to remember two marked numbers in a long sequence and add them. Specifically, the inputs consists of a two dimensional sequence . The first coordinate ] is uniformly sampled between 0 and 1, and the second coordinate is 0 at each save two; in these two entries, . The required output is the , where .

#### 3.2.1 a solution mechanism

This problem has a simple, explicit solution using a LT-RNN with a ReLU non-linearity and a one dimensional hidden state. Namely: set

, , , and . Then at each time step , if , then nothing is added to the hidden state, as . On the other hand, if , then exactly is added to .

This mechanism has been known (at least implicitly, although we don’t know if it has been written down explicitly before) at least since (Hochreiter & Schmidhuber, 1997), and it can be seen as a very simple LSTM model, with the following gates:

 i=xj[1]f=1o=1g=(xj[2]−1)+ (8)

and no non-linearity in Equation (5).

### 3.3 Comparison between the tasks

Note that the matrix

in the mechanism for the adding problem is the “identity”. We can build a more redundant solution by using a larger identity matrix. We can describe the identity using the same block structure as the matrix

defined for the copy task; namely each . On the other hand, the for the copy task acts as a “clock” that synchronizes after a fixed number of steps . It is important for the mechanism we described that the clock looks random at any time between and . For example, if we had instead used the same in each block , the mechanism would not succeed. The transition matrices for the addition task and the copy task are thus opposites in the sense that for addition, all the are the same (i.e. a mass on the unit circle), and for copy, the

are as uniformly distributed on the unit circle as possible.

In the experiments below, we will show that it is hard for an LT-RNN to learn the adding task when its transition matrix is initialized as a random orthogonal matrix but easy when initialized with the identity, and vice-versa for the copy task. One way to get a “unified” solution is to use pooling, as in 7. Then when initialized with a matrix with distributed uniformly, the decoder can choose to use the pooled hiddens (which through away the phase, and so appear identity-like) for the adding task, or use the raw hiddens, which are clock-like.

## 4 Experiments

### 4.1 Impact of Initialization

Based on the above analysis, we hypothesize that an LT-RNN with random orthogonal initialization (denoted LT-ORNN) should perform well on the sequence memorization problem, and an LT-RNN with identity initialization (denoted LT-IRNN) should perform well on the addition task. To test this, we conducted the following experiment on both the copy and addition task for different timescales. For each task and timescale, we trained 8 LT-ORNNs and 8 LT-RNNs with different random seeds. For LT-ORNNs, we initialized the transition matrix using a Gaussian distribution with mean 0 and variance

(where

is the number of hidden units) and then projected it to its nearest orthogonal matrix by setting its singular values to 1.

In all experiments, we used RMSProp to train our networks with a fixed learning rate and a decay rate of

. In preliminary experiments we tried different learning rates in and chose the largest one for which the loss did not diverge, for the LT-RNN’s we used .

We also included LSTMs in all our experiments as a baseline. We used the same method as for LT-RNN to pick the learning rate, and ended up with .

For all experiments, we normalized the gradients with respect to hidden activations by , where denotes the number of timesteps. In preliminary experiments, we also found that for LT-RNN models the activations frequently exploded whenever the largest singular value of the transition matrix became much greater than 1. Therefore, we adopted a simple activation clipping strategy where we rescaled activations to to have magnitude whenever their magnitude exceeded . In our experiments we chose .

Figure 2 shows the results on the copy task for the LSTM, LT-ORNN and LT-IRNN. All networks are trained with 80 hidden units. We see that the LSTM has difficulty beating the baseline performance of only outputting the empty symbol; however it does eventually converge to the solution (this is not shown in the figure). However, the LT-ORNN solves the task almost immediately. We note that this behavior is similar to that of the uRNN in (Martin Arjovsky, 2015), which is paramaterized in a way that makes it easy to recover the explicit solution described above. The LT-IRNN is never able to find the solution.

Figure 3 shows the results of the addition task for and timesteps. All networks are trained with 128 hidden units. For , we trained a single LT-ORNN and LT-IRNN due to time constraints. In contrast to the copy task, here the LT-IRNN is able to efficiently solve the problem whereas the LT-ORNN is only able to solve it after a very long time, or not at all. The LSTM is also able to easily solve the task, which is consistent with the original work of (Hochreiter & Schmidhuber, 1997) where the authors report solving the task for up to 1000 timesteps. We note that this LSTM baseline differs from that of (Martin Arjovsky, 2015; Quoc V. Le, 2015) where it is reported to have more difficulty solving the addition task. We hypothesize that this difference is due to the use of different variants of the LSTM architecture such as peephole connections.

### 4.2 Variable Length Copy Task

Having seen the stark impact of initialization on the performance of LT-IRNNs and LT-ORNNs for the copy and addition task, we then tested them on a problem for which we did not have a (roughly fixed size) solution mechanism, namely the variable length copy task. Figure 5 shows the performance of an LT-IRNN, LT-ORNN and LSTM (each with 80 hidden units) on the variable length copy task with timesteps. Even though the number of timesteps is significantly less than in other tasks, neither the LT-IRNN nor the LT-ORNN are able to beat the chance baseline, whereas the LSTM is able to solve the task even though its convergence is slow. This experiment is a classic example of how a detail of construction of a synthetic benchmark can favor a model in a way that fails to generalize to other tasks.

### 4.3 Pooling Experiments

We next ran a series of experiments to examine the effect of feeding pooled outputs to the decoder, to see if we could obtain good performance on both the copy and addition tasks with a single architecture and initialization. In these experiments, we added a soft penalty on the transition matrix

to keep it orthogonal throughout training. Specifically, at every iteration we applied one step of stochastic gradient descent to minimize the loss

, evaluated at random points on the unit sphere. Note that this requires operations and a regular update requires operations, so adding this soft constraint has negligible computational overhead. In our experiments we set , which was the same at the minibatch size.

In all pooling experiments we used a pool size and stride of 2. The results are shown in Figure

6. The LT-ORNN with pooling is easily able to solve the copy task for both timescales, and approximately solves the addition task for both timescales as well, even though convergence is slower than the LT-IRNN. Its success on the copy task is not surprising, since by zeroing out the matrix in Equation 7 it can solve the problem with the same solution as the regular LT-ORNN. The good performance on the adding task is somewhat more interesting. To gain insight into how the network stores information in a stable manner while having an (approximately) orthogonal transition matrix, we plotted the activations of its hidden states over time as it processes an input sequence. This is displayed in Figure 5. We observe relatively constant activations until the first marked number is encountered, which triggers oscillatory patterns along certain dimensions. When the second marked number is seen, existing oscillations are amplified and new ones emerge. This suggests that the network stores information stably through the radius of its hidden state’s rotations along different 2-dimensional subspaces. The information is then recovered as the phase is discarded though the pooling operation. Thus the model can have “uniform” clock-like oscillations that are perceived as -like after the pooling.

## 5 Conclusion

In this work, we analyzed two standard synthetic long-term memory problems and provided explicit RNN solutions for them. We found that the (fixed length ) copy problem can be solved with an RNN with a transition matrix that is a root of the identity matrix , and whose eigenvalues are well distributed on the unit circle, and we remarked that random orthogonal matrices almost satisfy this description. We also saw that the addition problem can be solved with as a transition matrix. We showed that correspondingly, initializing with allows a linear-transition RNN to easily be optimized for solving the addition task, and initializing with random orthogonal matrix allows easy optimization for the copy task; but that flipping these leads to poor results. Finally, we showed how one can use pooling to allow the model to make the decision between the two regimes.