Input-Output Equivalence of Unitary and Contractive RNNs

1 Introduction

Recurrent neural networks (RNNs) – originally proposed in the late 1980s Rumelhart et al. (1988); Elman (1990) – refer to a widely-used and powerful class of models for time series and sequential data. In recent years, RNNs have become particularly important in speech recognition Graves et al. (2013); Hinton et al. (2012)

and natural language processing

Collobert et al. (2011); Bahdanau et al. (2014); Sutskever et al. (2014) tasks.

A well-known challenge in training recurrent neural networks is the vanishing and exploding gradient problem Bengio et al. (1993); Pascanu et al. (2013). RNNs have a transition matrix that maps the hidden state at one time to the next time. When the transition matrix has an induced norm greater than one, the RNN may become unstable. In this case, small perturbations of the input at some time can result in a change in the output that grows exponentially over the subsequent time. This instability leads to a so-called exploding gradient. Conversely, when the norm is less than one, perturbations can decay exponentially so inputs at one time have negligible effect in the distant future. As a result, the loss surface associated with RNNs can have steep walls that may be difficult to minimize. Such problems are particularly acute in systems with long-term dependencies, where the output sequence can depend strongly on the input sequence many time steps in the past.

Unitary RNNs (URNNs) Arjovsky et al. (2016)

is a simple and commonly-used approach to mitigate the vanishing and exploding gradient problem. The basic idea is to restrict the transition matrix to be unitary (an orthogonal matrix for the real-valued case). The unitary transitional matrix is then combined with a non-expansive activation such as a ReLU or sigmoid. As a result, the overall transition mapping cannot amplify the hidden states, thereby eliminating the exploding gradient problem. In addition, since all the singular values of a unitary matrix equal 1, the transition matrix does not attenuate the hidden state, potentially mitigating the vanishing gradient problem as well. (Due to activation, the hidden state may still be attenuated). Some early work in URNNs suggested that they could be more effective than other methods, such as long short-term memory (LSTM) architectures and standard RNNs, for certain learning tasks involving long-term dependencies

Jing et al. (2017); Arjovsky et al. (2016) – see a short summary below.

Although URNNs may improve the stability of the network for the purpose of optimization, a basic issue with URNNs is that the unitary contraint may potentially reduce the set of input-output mappings that the network can model. This paper seeks to rigorously characterize how restrictive the unitary constraint is on an RNN. We evaluate this restriction by comparing the set of input-output mappings achievable with URNNs with the set of mappings from all RNNs. As described below, we restrict our attention to RNNs that are contractive in order to avoid unstable systems.

We show three key results:

Given any contractive RNN with $n$ hidden states and ReLU activations, there exists a URNN with at most $2 n$ hidden states and the identical input-ouput mapping.
This result is tight in the sense that, given any $n > 0$ , there exists at least one contractive RNN such that any URNN with the same input-output mapping must have at least $2 n$ states.
The equivalence of URNNs and RNNs depends on the activation. For example, we show that there exists a contractive RNN with sigmoid activations such that there is no URNN with any finite number of states that exactly matches the input-output mapping.

The implication of this result is that, for RNNs with ReLU activations, there is no loss in the expressiveness of model when imposing the unitary constraint. As we discuss below, the penalty is a two-fold increase in the number of parameters.

Of course, the expressiveness of a class of models is only one factor in their real performance. Based on these results alone, one cannot determine if URNNs will outperform RNNs in any particular task. Earlier works have found examples where URNNs offer some benefits over LSTMs and RNNs Arjovsky et al. (2016); Wisdom et al. (2016). But in the simulations below concerning modeling slowly-varying nonlinear dynamical systems, we see that URNNs with $2 n$ states perform approximately equally to RNNs with $n$ states.

Theoretical results on generalization error are an active subject area in deep neural networks. Some measures of model complexity such as Neyshabur et al. (2017) are related to the spectral norm of the transition matrices. For RNNs with non-contractive matrices, these complexity bounds will grow exponentially with the number of time steps. In contrast, since unitary matrices can bound the generalization error, this work can also relate to generalizability.

Prior work

The vanishing and exploding gradient problem in RNNs has been known almost as early as RNNs themselves Bengio et al. (1993); Pascanu et al. (2013)

. It is part of a larger problem of training models that can capture long-term dependencies, and several proposed methods address this issue. Most approaches use some form of gate vectors to control the information flow inside the hidden states, the most widely-used being LSTM networks

Hochreiter and Schmidhuber (1997). Other gated models include Highway networks Srivastava et al. (2015)

and gated recurrent units (GRUs)

Cho et al. (2014)

. L1/L2 penalization on gradient norms and gradient clipping were proposed to solve the exploding gradient problem in

Pascanu et al. (2013). With L1/L2 penalization, capturing long-term dependencies is still challenging since the regularization term quickly kills the information in the model. A more recent work Pennington et al. (2018) has successfully trained very deep networks by carefully adjusting the initial conditions to impose an approximate unitary structure of many layers.

Unitary evolution RNNs (URNNs) are a more recent approach first proposed in Arjovsky et al. (2016). Orthogonal constraints were also considered in the context of associative memories White et al. (2004). One of the technical difficulties is to efficiently parametrize the set of unitary matrices. The numerical simulations in this work focus on relatively small networks, where the parameterization is not a significant computational issue. Nevertheless, for larger numbers of hidden states, several approaches have been proposed. The model in Arjovsky et al. (2016)

parametrizes the transition matrix as a product of reflection, diagonal, permutation, and Fourier transform matrices. This model spans a subspace of the whole unitary space, thereby limiting the expressive power of RNNs. The work

Wisdom et al. (2016) overcomes this issue by optimizing over full-capacity unitary matrices. A key limitation in this work, however, is that the projection of weights on to the unitary space is not computationally efficient. A tunable, efficient parametrization of unitary matrices is proposed in Jing et al. (2017). This model provides the computational complexity of

O (1)

per parameter. The unitary matrix is represented as a product of rotation matrices and a diagonal matrix. By grouping specific rotation matrices, the model provides tunability of the span of the unitary space and enables using different capacities for different tasks. Combining the parametrization in Jing et al. (2017) for unitary matrices and the “forget” ability of the GRU structure, Cho et al. (2014); Jing et al. (2019) presented an architecture that outperforms conventional models in several long-term dependency tasks. Other methods such as orthogonal RNNs proposed by Mhammedi et al. (2017) showed that the unitary constraint is a special case of the orthogonal constraint. By representing an orthogonal matrix as a product of Householder reflectors, we are able span the entire space of orthogonal matrices. Imposing hard orthogonality constraints on the transition matrix limits the expressiveness of the model and speed of convergence and performance may degrade Vorontsov et al. (2017).

Figure 1: Recurrent Neural Network (RNN) model.

2 RNNs and Input-Output Equivalence

RNNs.

We consider recurrent neural networks (RNNs) representing sequence-to-sequence mappings of the form


	$h^{(k)} = ϕ (W h^{(k - 1)} + F x^{(k)} + b), h^{(- 1)} = h_{- 1},$		(1a)
	$y^{(k)} = C h^{(k)},$		(1b)

parameterized by $Θ = (W, F, b, C, h_{- 1})$ . The system is shown in Fig. 1. The system maps a sequence of inputs $x^{(k)} \in R^{m}$ , $k = 0, 1, \dots, T - 1$ to a sequence of outputs $y^{(k)} \in R^{p}$ . In equation (1), $ϕ$

is the activation function (e.g. sigmoid or ReLU);

h^{(k)} \in R^{n}

is an internal or hidden state;

W \in R^{n \times n}, F \in R^{n \times m}

, and

C \in R^{p \times n}

are the hidden-to-hidden, input-to-hidden, and hidden-to-output weight matrices respectively; and

b

is the bias vector. We have considered the initial condition,

h_{- 1}

, as part of the parameters, although we will often take

h_{- 1} = 0

. Given a set of parameters

Θ

, we will let

y = G (x, Θ)

(2)

denote the resulting sequence-to-sequence mapping. Note that the number of time samples, $T$ , is fixed throughout our discussion.

Recall Strang (1993) that a matrix $W$ is unitary if $W^{H} W = W W^{H} = I$ . When a unitary matrix is real-valued, it is also called orthogonal. In this work, we will restrict our attention to real-valued matrices, but still use the term unitary for consistency with the URNN literature. A Unitary RNN or URNN is simply an RNN (1) with a unitary state-to-state transition matrix $W$ . A key property of unitary matrices is that they are norm-preserving, meaning that $∥ W h^{(k)} ∥_{2} = ∥ h^{(k)} ∥_{2}$ . In the context of (1a), the unitary constraint implies that the transition matrix does not amplify the state.

Equivalence of RNNs.

Our goal is to understand the extent to which the unitary constraint in a URNN restricts the set of input-output mappings. To this end, we say that the RNNs for two parameters $Θ_{1}$ and $Θ_{2}$ are input-output equivalent if the sequence-to-sequence mappings are identical,

G (x, Θ_{1}) = G (x, Θ_{2}) for % all x = (x^{(0)}, \dots, x^{(T - 1)}) .

(3)

That is, for all input sequences $x$ , the two systems have the same output sequence. Note that the hidden internal states $h^{(k)}$ in the two systems may be different. We will also say that two RNNs are equivalent on a set of $X$ of inputs if (3) holds for all $x \in X$ .

It is important to recognize that input-output equivalence does not imply that the parameters $Θ_{1}$ and $Θ_{2}$ are identical. For example, consider the case of linear RNNs where the activation in (1) is the identity, $ϕ (z) = z$ . Then, for any invertible $T$ , the transformation

W \to T W T^{- 1}, C \to C T^{- 1}, F \to T F, h_{- 1} \to T h_{- 1},

(4)

results in the same input-output mapping. However, the internal states $h^{(k)}$ will be mapped to $T h^{(k)}$ . The fact that many parameters can lead to identical input-output mappings will be key to finding equivalent RNNs and URNNs.

Contractive RNNs.

The spectral norm Strang (1993) of a matrix $W$ is the maximum gain of the matrix $∥ W ∥ := {max}_{h \neq 0} \frac{∥ W h ∥_{2}}{∥ h ∥_{2}}$ . In an RNN (1), the spectral norm $∥ W ∥$ measures how much the transition matrix can amplify the hidden state. For URNNs, $∥ W ∥ = 1$ . We will say an RNN is contractive if $∥ W ∥ < 1$ , expansive if $∥ W ∥ > 1$ , and non-expansive if $∥ W ∥ \leq 1$ . In the sequel, we will restrict our attention to contractive and non-expansive RNNs. In general, given an expansive RNN, we cannot expect to find an equivalent URNN. For example, suppose $h^{(k)} = h^{(k)}$ is scalar. Then, the transition matrix $W$ is also scalar $W = w$ and $w$ is expansive if and only if $| w | > 1$ . Now suppose the activation is a ReLU $ϕ (h) = max {0, h}$ . Then, it is possible that a constant input $x^{(k)} = x_{0}$ can result in an output that grows exponentially with time: $y^{(k)} = c o n s t \times w^{k}$ . Such an exponential increase is not possible with a URNN. We consider only non-expansive RNNs in the remainder of the paper. Some of our results will also need the assumption that the activation function $ϕ (\cdot)$ in (1) is non-expansive:

This property is satisfied by the two most common activations, sigmoids and ReLUs.

Equivalence of Linear RNNs.

To get an intuition of equivalence, it is useful to briefly review the concept in the case of linear systems Kailath (1980). Linear systems are RNNs (1) in the special case where the activation function is identity, $ϕ (z) = z$ ; the initial condition is zero, $h_{- 1} = 0$ ; and the bias is zero, $b = 0$ . In this case, it is well-known that two systems are input-output equivalent if and only if they have the same transfer function,

H (s) := C (s I - W)^{- 1} F .

(5)

In the case of scalar inputs and outputs, $H (s)$ is a rational function of the complex variable $s$ with numerator and denominator degree of at most $n$ , the dimension of the hidden state $h^{(k)}$ . Any state-space system (1) that achieves a particular transfer function is called a realization of the transfer function. Hence two linear systems are equivalent if and only if they are the realizations of the same transfer function.

A realization is called minimal if it is not equivalent some linear system with fewer hidden states. A basic property of realizations of linear systems is that they are minimal if and only if they are controllable and observable. The formal definition is in any linear systems text, e.g. Kailath (1980). Loosely, controllable implies that all internal states can be reached with an appropriate input and observable implies that all hidden states can be observed from the ouptut. In absence of controllability and observability, some hidden states can be removed while maintaining input-output equivalence.

3 Equivalence Results for RNNs with ReLU Activations

Our first results consider contractive RNNs with ReLU activations. For the remainder of the section, we will restrict our attention to the case of zero initial conditions, $h^{(- 1)} = 0$ in (1).

Theorem 3.1

Let $y = G (x, Θ_{c})$ be a contractive RNN with ReLU activation and states of dimension $n$ . Fix $M > 0$ and let $X$ be the set of all sequences such that $∥ x^{(k)} ∥_{2} \leq M < \infty$ for all $k$ . Then there exists a URNN with state dimension $2 n$ and parameters $Θ_{u} = (W_{u}, F_{u}, b_{u}, C_{u})$ such that for all $x \in X$ , $G (x, Θ_{c}) = G (x, Θ_{u})$ . Hence the input-output mapping is matched for bounded inputs.

See Appendix A.

Theorem 3.1 shows that for any contractive RNN with ReLU activations, there exists a URNN with at most twice the number of hidden states and the identical input-output mapping. Thus, there is no loss in the set of input-output mappings with URNNs relative to general contractive RNNs on bounded inputs.

The penalty for using RNNs is the two-fold increase in state dimension, which in turn increases the number of parameters to be learned. We can estimate this increase in parameters as follows: The raw number of parameters for an RNN (

1) with

n

hidden states,

p

outputs and

m

inputs is

n^{2} + (p + m + 1) n

. However, for ReLU activations, the RNNs are equivalent under the transformations (4) using diagonal positive

T

. Hence, the number of degrees of freedom of a general RNN is at most

d_{r n n} = n^{2} + (p + m) n

. We can compare this value to a URNN with

2 n

hidden states. The set of

2 n \times 2 n

unitary

W

has

2 n (2 n - 1) / 2

degrees of freedom Stewart (1980). Hence, the total degrees of freedom in a URNN with

2 n

states is at most

d_{u r n n} = n (2 n - 1) + 2 n (p + m)

. We conclude that a URNN with

2 n

hidden states has slightly fewer than twice the number of parameters as an RNN with

n

hidden states.

We note that there are cases that the contractivity assumption is limiting, however, the limitations may not always be prohibitive. We will see in our experiments that imposing the contractivity constraint can improve learning for RNNs when models have sufficiently large numbers of time steps. Some related results where bounding the singular values help with the performance can be found in Vorontsov et al. (2017).

We next show a converse result.

Theorem 3.2

For every positive $n$ , there exists a contractive RNN with ReLU nonlinearity and state dimension $n$ such that every equivalent URNN has at least $2 n$ states.

See Appendix B.1 in the Supplementary Material.

The result shows that the $2 n$ achievability bound in Theorem 3.1 is tight, at least in the worst case. In addition, the RNN constructed in the proof of Theorem 3.2 is not particularly pathological. We will show in our simulations in Section 5 that URNNs typically need twice the number of hidden states to achieve comparable modeling error as an RNN.

4 Equivalence Results for RNNs with Sigmoid Activations

Equivalence between RNNs and URNNs depends on the particular activation. Our next result shows that with sigmoid activations, URNNs are, in general, never exactly equivalent to RNNs, even with an arbitrary number of states.

We need the following technical definition: Consider an RNN (1) with a standard sigmoid activation $ϕ (z) = 1 / (1 + e^{- z})$ . If $W$ is non-expansive, then a simple application of the contraction mapping principle shows that for any constant input $x^{(k)} = x^{*}$ , there is a fixed point in the hidden state $h^{*} = ϕ (W h^{*} + F x^{*} + b)$ . We will say that the RNN is controllable and observable at $x^{*}$ if the linearization of the RNN around $(x^{*}, h^{*})$ is controllable and observable.

Theorem 4.1

There exists a contractive RNN with sigmoid activation function $ϕ$ with the following property: If a URNN is controllable and observable at any point $x^{*}$ , then the URNN cannot be equivalent to the RNN for inputs $x$ in the neighborhood of $x^{*}$ .

See Appendix B.2 in the Supplementary Material.

The result provides a converse on equivalence: Contractive RNNs with sigmoid activations are not in general equivalent to URNNs, even if we allow the URNN to have an arbitrary number of hidden states. Of course, the approximation error between the URNN and RNN may go to zero as the URNN hidden dimension goes to infinity (e.g., similar to the approximation results in Funahashi and Nakamura (1993)). However, exact equivalence is not possible with sigmoid activations, unlike with ReLU activations. Thus, there is fundamental difference in equivalence for smooth and non-smooth activations.

We note that the fundamental distinction between Theorem 3.1 and the opposite result in Theorem 4.1

is that the activation is smooth with a positive slope. With such activations, you can linearize the system, and the eigenvalues of the transition matrix become visible in the input-output mapping. In contrast, ReLUs can zero out states and suppress these eigenvalues. This is a key insight of the paper and a further contribution in understanding nonlinear systems.

5 Numerical Simulations

In this section, we numerically compare the modeling ability of RNNs and URNNs where the true system is a contractive RNN with long-term dependencies. Specifically, we generate data from multiple instances of a synthetic RNN where the parameters in (1) are randomly generated. For the true system, we use $m = 2$ input units, $p = 2$ output units, and $n = 4$ hidden units at each time step. The matrices $F$ , $C$ and $b$ are generated as i.i.d. Gaussians. We use a random transition matrix,

W=I−ϵA\sf TA/∥A∥2,

(6)

where $A$ is Gaussian i.i.d. matrix and $ϵ$ is a small value, taken here to be $ϵ = 0.01$ . The matrix (6) will be contractive with singular values in $(1 - ϵ, 1)$ . By making $ϵ$ small, the states of the system will vary slowly, hence creating long-term dependencies. In analogy with linear systems, the time constant will be approximately $1 / ϵ = 100$ time steps. We use ReLU activations. To avoid degenerate cases where the outputs are always zero, the biases $b$ are adjusted to ensure that the each hidden state is on some target $60 %$ of the time using a similar procedure as in Fletcher et al. (2018).

The trials have $T = 1000$ time steps, which corresponds to 10 times the time constant $1 / ϵ = 100$

of the system. We added noise to the output of this system such that the signal-to-noise ratio (SNR) is 15 dB or 20 dB. In each trial, we generate 700 training samples and 300 test sequences from this system.

Given the input and the output data of this contractive RNN, we attempt to learn the system with: (i) standard RNNs, (ii) URNNs, and (iii) LSTMs. The hidden states in the model are varied in the range $n = [2, 4, 6, 8, 10, 12, 14]$ , which include values both above and below the true number of hidden states $n_{t r u e} = 4$

. We used mean-squared error as the loss function. Optimization is performed using Adam

Kingma and Ba (2014)

optimization with a batch size = 10 and learning rate = 0.01. All models are implemented in the Keras package in Tensorflow. The experiments are done over 30 realizations of the original contractive system.

For the URNN learning, of all the proposed algorithms for enforcing the unitary constraints on transition matrices during training Jing et al. (2017); Wisdom et al. (2016); Arjovsky et al. (2016); Mhammedi et al. (2017)

, we chose to project the transition matrix on the full space of unitary matrices after each iteration using singular value decomposition (SVD). Although SVD requires

O (n^{3})

computation for each projection, for our choices of hidden states it performed faster than the aforementioned methods.

Since we have training noise and since optimization algorithms can get stuck in local minima, we cannot expect “exact" equivalence between the learned model and true system as in the theorems. So, instead, we look at the test error as a measure of the closeness of the learned model to the true system. Figure 2 on the left shows the test $R^{2}$ for a Gaussian i.i.d. input and output with SNR = 20 dB for RNNs, URNNs, and LSTMs. The red dashed line corresponds to the optimal $R^{2}$ achievable at the given noise level.

Note that even though the true RNN has $n_{t r u e} = 4$ hidden states, the RNN model does not obtain the optimal test $R^{2}$ at $n = 4$ . This is not due to training noise, since the RNN is able to capture the full dynamics when we over-parametrize the system to $n \approx 8$ hidden states. The test error in the RNN at lower numbers of hidden states is likely due to the optimization being caught in a local minima.

What is important for this work though is to compare the URNN test error with that of the RNN. We observe that URNN requires approximately twice the number of hidden states to obtain the same test error as achieved by an RNN. To make this clear, the right plot shows the same performance data with number of states adjusted for URNN. Since our theory indicates that a URNN with $2 n$ hidden states is as powerful as an RNN with $n$ hidden states, we compare a URNN with $2 n$ hidden units directly with an RNN with $n$ hidden units. We call this the adjusted hidden units. We see that the URNN and RNN have similar test error when we appropriately scale the number of hidden units as predicted by the theory.

For completeness, the left plot in Figure 2 also shows the test error with an LSTM. It is important to note that the URNN has almost the same performance as an LSTM with considerably smaller number of parameters.

Figure 3 shows similar results for the same task with SNR = 15 dB. For this task, the input is sparse

Gaussian i.i.d., i.e. Gaussian with some probability

p = 0.02

and

0

with probability

1 - p

. The left plot shows the

R^{2}

vs. the number of hidden units for RNNs and URNNs and the right plot shows the same results once the number of hidden units for URNN is adjusted.

We also compared the modeling ability of URNNs and RNNs using the Pixel-Permuted MNIST task. Each MNIST image is a $28 \times 28$ grayscale image with a label between 0 and 9. A fixed random permutation is applied to the pixels and each pixel is fed to the network in each time step as the input and the output is the predicted label for each image Arjovsky et al. (2016); Jing et al. (2017); Vorontsov et al. (2017).

We evaluated various models on the Pixel-Permuted MNIST task using validation based early stopping. Without imposing a contractivity constraint during learning, the RNN is either unstable or requires a slow learning rate. Imposing a contractivity constraint improves the performance. Incidentally, using a URNN improves the performance further. Thus, contractivity can improve learning for RNNs when models have sufficiently large numbers of time steps.

Figure 2: Test $R^{2}$ on synthetic data for a Gaussian i.i.d. input and output SNR=20 dB.

Figure 3: Test $R^{2}$ on synthetic data for a Gaussian i.i.d. input and output SNR=15 dB.

Figure 4:
Accuracy on Permuted MNIST task for various models trained with RMSProp, validation-based early termination, and initial learning rate
lr. (1) URNN model: RNN model with unitary constraint; (2) ContRNN: RNN with a contractivity constraint; (3 & 4) RNN model with no contractivity or unitary constraint (two learning rates). We see contractivity improves performance, and unitary constraints improve performance further.

6 Conclusion

Several works empirically show that using unitary recurrent neural networks improves the stability and performance of the RNNs. In this work, we study how restrictive it is to use URNNs instead of RNNs. We show that URNNs are at least as powerful as contractive RNNs in modeling input-output mappings if enough hidden units are used. More specifically, for any contractive RNN we explicitly construct a URNN with twice the number of states of the RNN and identical input-output mapping. We also provide converse results for the number of state and the activation function needed for exact matching. We emphasize that although it has been shown that URNNs outperform standard RNNs and LSTM in many tasks that involve long-term dependencies, our main goal in this paper is to show that from an approximation viewpoint, URNNs are as expressive as general contractive RNNs. By a two-fold increase in the number of parameters, we can use the stability benefits they bring for optimization of neural networks.

Acknowledgements

The work of M. Emami, M. Sahraee-Ardakan, A. K. Fletcher was supported in part by the National Science Foundation under Grants 1254204 and 1738286, and the Office of Naval Research under Grant N00014-15-1-2677. S. Rangan was supported in part by the National Science Foundation under Grants 1116589, 1302336, and 1547332, NIST, the industrial affiliates of NYU WIRELESS, and the SRC.

Appendix A Proof of Theorem 3.1

The basic idea is to construct a URNN with $2 n$ states such that first $n$ states match the states of RNN and the last $n$ states are always zero. To this end, consider any contractive RNN,

h_{c}^{(k)} = ϕ (W_{c} h_{c}^{(k - 1)} + F_{c} x^{(k)} + b_{c}), y^{(k)} = C_{c} h_{c}^{(k)},

where $h^{(k)} \in R^{n}$ . Since $W$ is contractive, we have $∥ W ∥ \leq ρ$ for some $ρ < 1$ . Also, for a ReLU activation, $∥ ϕ (z) ∥ \leq ∥ z ∥$ for all pre-activation inputs $z$ . Hence,

	$∥ h_{c}^{(k)} ∥_{2}$	$= ∥ ϕ (W_{c} h_{c}^{(k - 1)} + F_{c} x^{(k)} + b_{c}) ∥_{2} \leq ∥ W_{c} h_{c}^{(k - 1)} + F_{c} x^{(k)} + b_{c} ∥_{2}$
		$\leq ρ ∥ h_{c}^{(k - 1)} ∥_{2} + ∥ F_{c} ∥ ∥ x^{(k)} ∥_{2} + ∥ b_{c} ∥_{2} .$

Therefore, with bounded inputs, $∥ x^{(k)} ∥ \leq M$ , we have the state is bounded,

∥ h^{(k)} ∥_{2} \leq \frac{1}{1 - ρ} [∥ F_{c} ∥ M + ∥ b_{c} ∥_{2}] =: M_{h} .

(7)

We construct a URNN as,

h_{u}^{(k)} = ϕ (W_{u} h_{u}^{(k - 1)} + F_{u} x^{(k)} + b_{u}), y^{(k)} = C_{u} h_{u}^{(k)}

where the parameters are of the form,

(8)

Let $W_{1} = W_{c}$ . Since $∥ W_{c} ∥ < 1$ , we have $I−W\sf TcWc⪰0$ . Therefore, there exists $W_{3}$ such that . With this choice of $W_{3}$ , the first $n$ columns of $W_{u}$ are orthonormal. Let $[\begin{matrix} W_{2} W_{4} \end{matrix}]$ extend these to an orthonormal basis for $R^{2 n}$ . Then, the matrix $W_{u}$ will be orthonormal.

Next, let $b_{2} = - M_{h} 1_{n \times 1}$ , where $M_{h}$ is defined in (7). We show by induction that for all $k$ ,

h_{1}^{(k)} = h_{c}^{(k)}, h_{2}^{(k)} = 0 .

(9)

If both systems are initialized at zero, (9) is satisfied at $k = - 1$ . Now, suppose this holds up to time $k - 1$ . Then,

	$h_{1}^{(k)}$	$= ϕ (W_{1} h_{1}^{(k - 1)} + W_{2} h_{2}^{(k - 1)} + F_{c} x^{(k)} + b_{c})$
		$= ϕ (W_{1} h_{1}^{(k - 1)} + F_{c} x^{(k)} + b_{c}) = h_{c}^{(k)},$

where we have used the induction hypothesis that $h_{2}^{(k - 1)} = 0$ . For $h_{2}^{(k)}$ , note that

∥ W_{3} h_{1}^{(k - 1)} ∥_{\infty} \leq ∥ W_{3} h_{1}^{(k - 1)} ∥_{2} \leq ∥ h_{1}^{(k - 1)} ∥ \leq M_{h},

(10)

where the last step follows from (7). Therefore,

W_{3} h_{1}^{(k - 1)} + W_{4} h_{2}^{(k - 1)} + b_{2} = W_{3} h_{1}^{(k - 1)} - M 1_{n \times 1} \leq 0 .

(11)

Hence with ReLU activation $h_{2}^{(k)} = ϕ (W_{3} h_{1}^{(k - 1)} + W_{4} h_{2}^{(k - 1)} + b_{2}) = 0$ . By induction, (9) holds for all $k$ . Then, if we define $C_{u} = [C_{c} 0]$ , we have the output of the URNN and RNN systems are identical

y_{u}^{(k)} = C_{u} h_{u}^{(k)} = C_{c} h_{1}^{(k)} = y_{c}^{(k)} .

This shows that the systems are equivalent.

References

[1] M. Arjovsky, A. Shah, and Y. Bengio (2016) Unitary evolution recurrent neural networks. In
International Conference on Machine Learning
, pp. 1120–1128. Cited by: §1, §1, §1, §5, §5.
[2] D. Bahdanau, K. Cho, and Y. Bengio (2014) Neural machine translation by jointly learning to align and translate. arXiv, pp. arXiv–1409. Cited by: §1.
[3] Y. Bengio, P. Frasconi, and P. Simard (1993) The problem of learning long-term dependencies in recurrent networks. In IEEE International Conference on Neural Networks, pp. 1183–1188. Cited by: §1, §1.
[4] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio (2014) On the properties of neural machine translation: encoder–decoder approaches. Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. External Links: Link, Document Cited by: §1, §1.
[5] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa (2011) Natural language processing (almost) from scratch. Journal of Machine Learning Research 12 (Aug), pp. 2493–2537. Cited by: §1.
[6] J. L. Elman (1990) Finding structure in time. Cognitive Science 14 (2), pp. 179–211. Cited by: §1.
[7] A. K. Fletcher, S. Rangan, and P. Schniter (2018) Inference in deep networks in high dimensions. In Proc. IEEE International Symposium on Information Theory, pp. 1884–1888. Cited by: §5.
[8] K. Funahashi and Y. Nakamura (1993) Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (6), pp. 801–806. Cited by: §4.
[9] A. Graves, A. Mohamed, and G. Hinton (2013) Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649. Cited by: §1.
[10] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury (2012) Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine 29. Cited by: §1.
[11] S. Hochreiter and J. Schmidhuber (1997) Long short-term memory. Neural Computation 9 (8), pp. 1735–1780. Cited by: §1.
[12] L. Jing, C. Gulcehre, J. Peurifoy, Y. Shen, M. Tegmark, M. Soljacic, and Y. Bengio (2019) Gated orthogonal recurrent units: on learning to forget. Neural Computation 31 (4), pp. 765–783. Cited by: §1.
[13] L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić (2017) Tunable efficient unitary neural networks (eunn) and their application to rnns. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1733–1741. Cited by: §1, §1, §5, §5.
[14] T. Kailath (1980) Linear systems. Vol. 156, Prentice-Hall Englewood Cliffs, NJ. Cited by: §2, §2.
[15] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §5.
[16] Z. Mhammedi, A. Hellicar, A. Rahman, and J. Bailey (2017) Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2401–2409. Cited by: §1, §5.
[17] B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro (2017)
Exploring generalization in deep learning
. In Advances in Neural Information Processing Systems, pp. 5947–5956. Cited by: §1.
[18] R. Pascanu, T. Mikolov, and Y. Bengio (2013) On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310–1318. Cited by: §1, §1.
[19] J. Pennington, S. S. Schoenholz, and S. Ganguli (2018) The emergence of spectral universality in deep networks. arXiv preprint arXiv:1802.09979. Cited by: §1.
[20] D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1988) Learning representations by back-propagating errors. Cognitive Modeling 5 (3), pp. 1. Cited by: §1.
[21] R. K. Srivastava, K. Greff, and J. Schmidhuber (2015) Highway networks. arXiv preprint arXiv:1505.00387. Cited by: §1.
[22] G. W. Stewart (1980) The efficient generation of random orthogonal matrices with an application to condition estimators. SIAM Journal on Numerical Analysis 17 (3), pp. 403–409. Cited by: §3.
[23] G. Strang (1993) Introduction to linear algebra. Vol. 3, Wellesley-Cambridge Press Wellesley, MA. Cited by: §2, §2.
[24] I. Sutskever, O. Vinyals, and Q. V. Le (2014) Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112. Cited by: §1.
[25] M. Vidyasagar (2002) Nonlinear systems analysis. Vol. 42, Siam. Cited by: §B.2.
[26] E. Vorontsov, C. Trabelsi, S. Kadoury, and C. Pal (2017) On orthogonality and learning recurrent networks with long term dependencies. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3570–3578. Cited by: §1, §3, §5.
[27] O. L. White, D. D. Lee, and H. Sompolinsky (2004) Short-term memory in orthogonal neural networks. Physical review letters 92 (14), pp. 148102. Cited by: §1.
[28] S. Wisdom, T. Powers, J. Hershey, J. Le Roux, and L. Atlas (2016) Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 4880–4888. Cited by: §1, §1, §5.

Appendix B Converse Theorem Proofs

b.1 Proof of Theorem 3.2

First consider the case when $n = 1$ with scalar inputs and outputs. Let $θ_{c} = (w_{c}, f_{c}, b_{c}, c_{c})$ be the parameters of a contractive RNN with $f_{c} = c_{c} = 1$ , $b_{c} = 0$ and $w_{c} \in (0, 1)$ . Hence, the contractive RNN is given by

h_{c}^{(k)} = ϕ (w_{c} h_{c}^{(k - 1)} + x^{(k)}), y^{(k)} = h_{c}^{(k)},

(12)

and $ϕ (z) = max {0, z}$ is the ReLU activation. Suppose $Θ_{u}$ are the parameters of an equivalent URNN. If $Θ$ has less than $2 n = 2$ states, it must have $n = 1$ state. Let the equivalent URNN be

h_{u}^{(k)}

= ϕ (w_{u} h_{u}^{(k - 1)} + f_{u} x^{(k)} + b_{u}), y^{(k)} = c_{u} h_{u}^{(k)},

(13)

for some parameters $Θ_{u} = (w_{u}, f_{u}, b_{u}, c_{u})$ . Since $w_{u}$ is orthogonal, either $w_{u} = 1$ or $w_{u} = - 1$ . Also, either $f_{u} > 0$ or $f_{u} < 0$ . First, consider the case when $w_{u} = 1$ and $f_{u} > 0$ . Then, there exists a large enough input $x^{(k)}$ such that for all time steps $k$ , both systems are operating in the active phase of ReLU. Therefore, we have two equivalent linear systems,

	$contractive RNN: h_{c}^{(k)}$	$= w_{c} h_{c}^{(k - 1)} + x^{(k)}, y^{(k)} = h_{c}^{(k)}$		(14)
	$URNN: h_{u}^{(k)}$	$= h_{u}^{(k - 1)} + f_{u} x^{(k)} + b_{u}, y^{(k)} = c_{u} h_{u}^{(k)} .$		(15)

In order to have identical input-output mapping for these linear systems for all $x$ , it is required that $w_{c} = 1$ , which is a contradiction. The other cases $w_{c} = - 1$ and $f_{u} < 0$ can be treated similarly. Therefore, at least $n = 2$ states are needed for the URNN to match the contractive RNN with $n = 1$ state.

For the case of general $n$ , consider the contractive RNN,

h^{(k)} = ϕ (W h^{(k - 1)} + F x^{(k)} + b), y^{(k)} = C h^{(k)},

(16)

where $W = d i a g (w_{c}, w_{c}, . . ., w_{c})$ , $F = d i a g (f_{c}, f_{c}, . . ., f_{c})$ , $b = b_{c} 1_{n \times 1}$ , and $C = d i a g (c_{c}, c_{c}, . . ., c_{c})$ . This system is separable in that if $y = G (x)$ then $y_{i} = G (x_{i}, θ_{c})$ for each input $i$ . A URNN system will need 2 states for each scalar system requiring a total of $2 n$ states.

b.2 Proof of Theorem 4.1

We use the same scalar contractive RNN (12), but with a sigmoid activation $ϕ (z) = 1 / (1 + e^{- z})$ . Let $Θ = (W_{u}, f_{u}, c_{u}, b_{u})$ be the parameters of any URNN with scalar input and outputs. Suppose the URNN is controllable and observable at an input value $x^{*}$ . Let $h_{c}^{*}$ and $h_{u}^{*}$ be, respectively, the fixed points of the hidden states for the contractive RNN and URNN:

	$contractive RNN: h_{c}^{*}$	$= ϕ (w_{c} h_{c} + x^{*}),$		(17)
	$URNN: h_{u}^{*}$	$= ϕ (W_{u} h_{u} + f_{u} x^{*} + b_{u}) .$		(18)

We take the linearizations [25] of each system around its fixed point and apply a small perturbation $Δ x$ around $x^{*}$ . Therefore, we have two linear systems with identical input-output mapping given by,

	$contractive RNN: Δ h_{c}^{(k)}$	$= d_{c} (w_{c} Δ h^{(k - 1)} + Δ x^{(k)}), y^{(k)} = Δ h_{c}^{(k)} + h_{c}^{*},$		(19)
	$URNN: Δ h_{u}^{(k)}$	$=Du(WuΔh(k−1)u+f\sf TuΔx(k)),y(k)=c\sf TuΔhu+c\sf Tuh∗u,$		(20)

where

d_{c} = ϕ^{'} (z_{c}^{*} = w_{c} h_{c}^{*} + x^{*}), D_{u} = ϕ^{'} (W_{u} h_{u}^{*} + f_{u} x^{*} + b_{u}),

are the derivatives of the activations at the fixed points. Since both systems are controllable and observable, their dimensions must be the same and the eigenvalues of the transition matrix must match. In particular, the URNN must be scalar, so $W_{u} = w_{u}$ for some scalar $w_{u}$ . For orthogonality, either $w_{u} = 1$ or $w_{u} = - 1$ . We look at the $w_{u} = 1$ case; the $w_{u} = - 1$ case is similar. Since the eigenvalues of the transition matrix must match we have,

d_{c} w_{c} = d_{u} \Rightarrow ϕ^{'} (w_{c} h_{c}^{*} + x^{*}) w_{c} = ϕ^{'} (h_{u}^{*} + f_{u} x^{*} + b_{u}) .

(21)

where $h_{u}^{*}$ and $h_{c}^{*}$ are the solutions to the fixed point equations:

h_{c}^{*} = ϕ (w_{c} h_{c} + x^{*}), h_{u}^{*} = ϕ (h_{u}^{*} + f_{b} x^{*} + b_{u}) .

(22)

Also, since two systems have the same output,

h_{c}^{*} = c_{u} h_{u}^{*} .

(23)

Now, (21) must hold at any input $x^{*}$ where the URNN is controllable and observable. If the URNN is controllable and observable at some $x^{*}$ , it is is controllable and observable in a neighborhood of $x^{*}$ . Hence, (21) and (23) holds in some neighborhood of $x^{*}$ . To write this mathematically, define the functions,

g_{c} (x^{*}) := [\begin{matrix} w_{c} ϕ^{'} (w_{c} h_{c}^{*} + x^{*}) h_{c}^{*} \end{matrix}], g_{u} (x^{*}) := [\begin{matrix} ϕ^{'} (h_{u}^{*} + f_{u} x^{*} + b_{u}) c_{u} h_{u}^{*} \end{matrix}],

(24)

where, for a given $x^{*}$ , $h_{u}^{*}$ and $h_{c}^{*}$ are the solutions to the fixed point equations (22). We must have that $g_{c} (x^{*}) = g_{u}^{*} (x^{*})$ for all $x^{*}$ in some neighborhood. Taking derivatives of (24) and using the fact that $ϕ (z)$ being a sigmoid, one can show that this matching can only occur when,

w_{c} = 1, b_{u} = 0, c_{u} = 1.

This is a contradiction since we have assumed that the RNN system is contractive which requires $| w_{c} | = 1$ .

Input-Output Equivalence of Unitary and Contractive RNNs

Authors

Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Training Input-Output Recurrent Neural Networks through Spectral Methods

Universal Approximation of Input-Output Maps by Temporal Convolutional Nets

Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics

Neural Speed Reading via Skim-RNN

Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations

Twin Networks: Using the Future as a Regularizer

1 Introduction

Prior work

2 RNNs and Input-Output Equivalence

RNNs.

Equivalence of RNNs.

Contractive RNNs.

Equivalence of Linear RNNs.

3 Equivalence Results for RNNs with ReLU Activations

Theorem 3.1

Theorem 3.2

4 Equivalence Results for RNNs with Sigmoid Activations

Theorem 4.1

5 Numerical Simulations

6 Conclusion

Acknowledgements

Appendix A Proof of Theorem 3.1

References

Appendix B Converse Theorem Proofs

b.1 Proof of Theorem 3.2

b.2 Proof of Theorem 4.1

Input-Output Equivalence of Unitary and Contractive RNNs

Authors

Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Training Input-Output Recurrent Neural Networks through Spectral Methods

Universal Approximation of Input-Output Maps by Temporal Convolutional Nets

Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics

Neural Speed Reading via Skim-RNN

Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations

Twin Networks: Using the Future as a Regularizer

This week in AI

1 Introduction

Prior work

2 RNNs and Input-Output Equivalence

RNNs.

Equivalence of RNNs.

Contractive RNNs.

Equivalence of Linear RNNs.

3 Equivalence Results for RNNs with ReLU Activations

Theorem 3.1

Theorem 3.2

4 Equivalence Results for RNNs with Sigmoid Activations

Theorem 4.1

5 Numerical Simulations

6 Conclusion

Acknowledgements

Appendix A Proof of Theorem 3.1

References

Appendix B Converse Theorem Proofs

b.1 Proof of Theorem 3.2

b.2 Proof of Theorem 4.1