Universality and approximation bounds for echo state networks with random weights

1 Introduction

Reservoir computing (Lukoševičius and Jaeger, 2009), such as echo state networks (Jaeger and Haas, 2004) and liquid state machines (Maass et al., 2002)

, is a paradigm for supervised learning of dynamical systems, which transforms input data into a high-dimensional space by a state-space nonlinear system called reservoir, and performs the learning task only on the readout. Due to the simplicity of this computational framework, it has been applied to many fields and made remarkable success in many tasks such as temporal pattern prediction, classification and generation

(Tanaka et al., 2019). Motivated by these empirical success, researchers have devoted a lot of efforts to theoretically understanding the properties and performances of reservoir computing models (see, for instance, Jaeger (2001); Buehner and Young (2006); Lukoševičius and Jaeger (2009); Yildiz et al. (2012); Grigoryeva and Ortega (2019); Gonon et al. (2020a, b) and references therein).

In particular, recent studies (Grigoryeva and Ortega, 2018, 2018; Gonon and Ortega, 2020, 2021)

showed that echo state networks are universal in the sense that they can approximate sufficiently regular input/output systems (i.e. operators) in various settings. However, these universality results do not guarantee the important property of echo state networks (and reservoir computing) that the state-space system is randomly generated, which is the major difference between reservoir computing and recurrent neural networks. To be concrete, consider the echo state network

{\begin{matrix} s_{t} = ϕ (W_{1} s_{t - 1} + W_{2} u_{t} + b), y_{t} = a \cdot s_{t}, \end{matrix}

(1.1)

where $u_{t}, y_{t}, s_{t}$ are the input, output and hidden state at time step $t$ , and $ϕ$ is a prescribed activation function. In general, the weight matrices $W_{1}, W_{2}, b$ are randomly generated and only the readout $a$ is trained in a supervised learning manner. But the universality results in the mentioned papers require that all the weights depend on the target system that we want to approximate. Hence, these results can not completely explain the approximation capacity of echo state networks. To overcome this drawback of current theories, the recent work of Gonon et al. (2020a)

studied the approximation of random neural networks in a Hilbert space setting and proposed a sampling procedure for the internal weights

W_{1}, W_{2}, b

so that echo state network (1.1) with ReLU activation is universal in

L^{2}

sense.

In this paper, we generalize these results and study the universality of echo state networks with randomly generated internal weights. Our main result gives a sufficient condition for the activation function $ϕ$ and the sampling procedure of $W_{1}, W_{2}, b$ so that the system (1.1) can approximate any continuous time-invariant operator in $L^{\infty}$ sense with high probability. In particular, for ReLU activation, we show that the echo state network (1.1) is universal when $W_{1} = c_{μ} W P W^{⊺}$ and $W_{2} = W Q$ , where $(W, b)$

is a random matrix whose entries are sampled independently from a general symmetric distribution

μ

c_{μ}

is a constant depending on

μ

and

P, Q

are fixed matrices defined by (3.5). Comparing with Gonon et al. (2020a), which takes advantage of the property of ReLU function to construct the reservoir, our construction makes use of the concentration of probability measure, and hence it can be applied to general activation functions.

1.1 Notations

We denote $N := {1, 2, \dots}$ as the set of positive integers. Let $Z$ (respectively, $Z_{+}$ and $Z_{-}$ ) be the set of integers (respectively, the non-negative and non-positive integers). The ReLU function is denoted by $σ (x) := max {x, 0}$ . Throughout the paper, $R^{d}$ is equipped with the sup-norm $∥ \cdot ∥_{\infty}$ , unless it is explicitly stated. For any $U \subseteq R^{d}$ , we use $U^{Z}$ to denote the set of sequences of the form $u = (\dots, u_{- 1}, u_{0}, u_{1}, \dots)$ , $u_{t} \in U$ , $t \in Z$ . The set $U^{Z_{+}}$ and $U^{Z_{-}}$ consist of right and left infinite sequences of the form $(u_{0}, u_{1}, \dots)$ and $(\dots, u_{- 1}, u_{0})$ , respectively. For any $i \leq j$ and $u \in U^{Z}$ , we denote $u_{i : j} := (u_{i}, \dots, u_{j}) \in U^{j - i + 1}$ and use the convention that, when $i = - \infty$ or $j = \infty$ , it denotes the left or right infinite sequence. And we will often regard $u_{- \infty : j}$ as an element of $U^{Z_{-}}$ . The supremum norm of the sequence is denoted by $∥ u ∥_{\infty} := {sup}_{t \in Z} ∥ u_{t} ∥_{\infty}$ . A weighting sequence $η : Z_{+} \to (0, 1]$ is a decreasing sequence with zero limit. The weight norm on $(R^{d})^{Z_{-}}$ associated to $η$ is denoted by $∥ u ∥_{η} := {sup}_{t \in Z_{-}} η_{- t} ∥ u_{t} ∥_{\infty}$ .

2 Continuous causal time-invariant operators

We study the uniform approximation problem for input/output systems or operators of signals in discrete time setting. We mainly consider the operators $F : ([- 1, 1]^{d})^{Z} \to R^{Z}$ that satisfy the following three properties:

[label=(0),parsep=0pt]
Causality: For any $u, v \in ([- 1, 1]^{d})^{Z}$ and $t \in Z$ , $u_{- \infty : t} = v_{- \infty : t}$ implies $F (u)_{t} = F (v)_{t}$ .
Time-invariance: $F (T_{τ} (u))_{t} = F (u)_{t - τ}$ for any $τ \in Z$ , where $T_{τ} : (R^{d})^{Z} \to (R^{d})^{Z}$ is time delay operator defined by $T_{τ} (u)_{t} = u_{t - τ}$ .
Continuity (fading memory property): $F : ([- 1, 1]^{d})^{Z} \to [- B, B]^{Z}$ is uniformly bounded for some $B > 0$ and is continuous with respect to the product topology.

Grigoryeva and Ortega (2018) gave a comprehensive study of these operators. We recall that the causal time-invariant operator $F : ([- 1, 1]^{d})^{Z} \to R^{Z}$ is one-to-one correspondence with the functional $F^{*} : ([- 1, 1]^{d})^{Z_{-}} \to R$ defined by $F^{*} (u_{-}) := F (u)_{0}$ where $u \in ([- 1, 1]^{d})^{Z}$ is any extension of $u_{-} \in ([- 1, 1]^{d})^{Z_{-}}$ such that $u_{- \infty : 0} = u_{-}$ ( $F^{*}$ is well-defined by causality). And we can reconstruct $F$ from $F^{*}$ by $F (u)_{t} = F^{*} (P_{Z_{-}} \circ T_{- t} (u))$ , where $P_{Z_{-}} : (R^{d})^{Z} \to (R^{d})^{Z_{-}}$ is the natural projection. In particular, for any causal time-invariant operators $F, G : ([- 1, 1]^{d})^{Z} \to R^{Z}$ ,

sup u \in ([- 1, 1]^{d})^{Z} ∥ F (u) - G (u) ∥_{\infty} = sup u_{-} \in ([- 1, 1]^{d})^{Z_{-}} | F^{*} (u_{-}) - G^{*} (u_{-}) | .

Hence, the approximation problem of causal time-invariant operators $F$ can be reduced to the approximation of functionals $F^{*}$ on $([- 1, 1]^{d})^{Z_{-}}$ .

We remark that the product topology on $(R^{d})^{Z}$ is different from the uniform topology induced by the sup-norm. However, for the set $([- B, B]^{d})^{Z_{-}}$ of uniformly bounded sequences, the product topology coincides with the topology induced by the weight norm $∥ u ∥_{η}$ of any weighting sequence $η$ . It is shown by Grigoryeva and Ortega (2018) that the causal time-invariant operator $F$ is continuous (i.e. has the fading memory property) if and only if the corresponding functional $F^{*}$ is continuous with respect to the product topology, which is equivalent to $F^{*}$ has fading memory property with respect to some (and hence any) weighting sequence $η$ , i.e. $F^{*}$ is a continuous function on the compact metric space $(([- 1, 1]^{d})^{Z_{-}}, ∥ \cdot ∥_{η})$ . In other words, for any $ϵ > 0$ , there exists a $δ (ϵ) > 0$ such that for any $u, v \in ([- 1, 1]^{d})^{Z_{-}}$ ,

∥ u - v ∥_{η} = sup t \in Z_{-} η_{- t} ∥ u_{t} - v_{t} ∥_{\infty} < δ (ϵ) implies | F^{*} (u) - F^{*} (v) | < ϵ .

Next, we introduce several notations to quantify the regularity of causal time-invariant operators. For any $m \in N$ , we can associate a function $F_{m}^{*} : ([- 1, 1]^{d})^{m} \to R$ to any causal time-invariant operator $F : ([- 1, 1]^{d})^{Z} \to R^{Z}$ by

F_{m}^{*} (u_{- m + 1}, \dots, u_{0}) := F^{*} (\dots, 0, u_{- m + 1}, \dots, u_{0}) .

(2.1)

If the functional $F^{*}$ can be approximated by $F_{m}^{*}$ arbitrarily well, we say $F$ has approximately finite memory. The following definition quantifies this approximation.

Definition 2.1 (Approximately finite memory).

For any causal time-invariant operator $F : ([- 1, 1]^{d})^{Z} \to R^{Z}$ , let $ϵ \geq 0$ and $m \in N$ , we denote

	$E_{F} (m)$	$:= sup u \in ([- 1, 1]^{d})^{Z_{-}} \| F^{} (u) - F_{m}^{} (u_{- m + 1 : 0}) \|,$
	$m_{F} (ϵ)$	$:= min {m \in N : E_{F} (m) \leq ϵ} .$

If $m_{F} (ϵ) < \infty$ for all $ϵ > 0$ , we say $F$ has approximately finite memory. If $m_{F} (0) < \infty$ , we say $F$ has finite memory.

Note that $m_{F} (ϵ)$ is a non-increasing function of $ϵ$ . By the time-invariance of $F$ , for any $t \in Z$ ,

sup u \in ([- 1, 1]^{d})^{Z} | F (u)_{t} - F_{m}^{*} (u_{t - m + 1 : t}) | = sup u \in ([- 1, 1]^{d})^{Z} | F^{*} (u_{- \infty : t}) - F_{m}^{*} (u_{t - m + 1 : t}) | = E_{F} (m) .

(2.2)

Hence, $E_{F} (m)$ quantifies how well $F$ can be approximated by functionals with finite memory.

Definition 2.2 (Modulus of continuity).

If $F^{*} : ([- 1, 1]^{d})^{Z_{-}} \to R$ has fading memory property with respect to some weighting sequence $η$ . We denote the modulus of continuity with respect to $∥ \cdot ∥_{η}$ by

ω_{F^{*}} (δ; η) := sup {| F^{*} (u) - F^{*} (u^{'}) | : u, u^{'} \in ([- 1, 1]^{d})^{Z_{-}}, ∥ u - u^{'} ∥_{η} \leq δ},

and the inverse modulus of continuity

ω_{F^{*}}^{- 1} (ϵ; η) := sup {δ > 0 : ω_{F^{*}} (δ; η) \leq ϵ} .

Similarly, the modulus of continuity (with respect to $∥ \cdot ∥_{\infty}$ ) of a continuous function $f : [- 1, 1]^{m} \to R$ and the inverse modulus of continuity are defined by

	$ω_{f} (δ)$	$:= sup {\| f (x) - f (x^{'}) \| : x, x^{'} \in [- 1, 1]^{m}, ∥ x - x^{'} ∥_{\infty} \leq δ},$
	$ω_{f}^{- 1} (ϵ)$	$:= sup {δ > 0 : ω_{f} (δ) \leq ϵ} .$

The next proposition quantifies the continuity of causal time-invariant operators by the approximately finite memory and modulus of continuity. This proposition is a modification of similar result in Hanson and Raginsky (2019) to our setting. It shows that a causal time-invariant operator $F$ is continuous if and only if it has approximately finite memory and each $F_{m}^{*}$ is continuous.

Proposition 2.3.

Let $F : ([- 1, 1]^{d})^{Z} \to R^{Z}$ be a causal time-invariant operator.

[label=(0),parsep=0pt]
If $F$ has approximately finite memory and $ω_{F_{m}^{*}} (δ) < \infty$ for any $m \in N$ and $δ > 0$ , then $F^{*}$ has fading memory property with respect to any weighting sequence $η$ : for any $ϵ > 0$ , $u, v \in ([- 1, 1]^{d})^{Z_{-}}$ and $m = m_{F} (ϵ / 3)$ ,

$∥ u - v ∥_{η} \leq η_{m - 1} ω_{F_{m}^{*}}^{- 1} (ϵ / 3) implies | F^{*} (u) - F^{*} (v) | \leq ϵ .$

In other words, $ω_{F_{*}} (η_{m - 1} ω_{F_{m}^{*}}^{- 1} (ϵ / 3); η) \leq ϵ$ for $m = m_{F} (ϵ / 3)$ .
If $F^{*}$ has fading memory property with respect to some weighting sequence $η$ , then $F$ has approximately finite memory with $E_{F} (m) \leq ω_{F^{*}} (η_{m}, η)$ ,

$m_{F} (ϵ) \leq min {m \in N : η_{m} \leq ω_{F^{*}}^{- 1} (ϵ; η)},$

and $ω_{F_{m}^{*}} (δ) \leq ω_{F^{*}} (δ; η)$ for any $m \in N$ .

Proof.

For the first part, by the definition of $m = m_{F} (ϵ / 3)$ ,

sup u \in ([- 1, 1]^{d})^{Z_{-}} | F^{*} (u) - F_{m}^{*} (u_{- m + 1 : 0}) | \leq ϵ / 3.

Since the weighting sequence is decreasing,

∥ u_{- m + 1 : 0} - v_{- m + 1 : 0} ∥_{\infty} \leq η_{m - 1}^{- 1} ∥ u - v ∥_{η} \leq ω_{F_{m}^{*}}^{- 1} (ϵ / 3),

which implies $| F_{m}^{*} (u_{- m + 1 : 0}) - F_{m}^{*} (v_{- m + 1 : 0}) | \leq ϵ / 3$ . Therefore,

		$\| F^{} (u) - F^{} (v) \|$
	$\leq$	$\| F^{} (u) - F_{m}^{} (u_{- m + 1 : 0}) \| + \| F_{m}^{} (u_{- m + 1 : 0}) - F_{m}^{} (v_{- m + 1 : 0}) \| + \| F_{m}^{} (v_{- m + 1 : 0}) - F^{} (v) \|$
	$\leq$	$ϵ / 3 + ϵ / 3 + ϵ / 3 = ϵ .$

For the second part, we observe that, for any $m \in N$ and $u \in ([- 1, 1]^{d})^{Z_{-}}$ ,

∥ u - (\dots, 0, u_{- m + 1 : 0}) ∥_{η} = sup t \leq - m η_{- t} ∥ u_{t} ∥_{\infty} \leq η_{m} .

Then, by the definition of $ω_{F^{*}} (δ; η)$ ,

E_{F} (m) = sup u \in ([- 1, 1]^{d})^{Z_{-}} | F^{*} (u) - F^{*} (\dots, 0, u_{- m + 1 : 0}) | \leq ω_{F^{*}} (η_{m}, η) .

If $m$ satisfies $η_{m} \leq ω_{F^{*}}^{- 1} (ϵ; η)$ , then $E_{F} (m) \leq ω_{F^{*}} (η_{m}, η) \leq ϵ$ . Hence,

m_{F} (ϵ) = min {m \in N : E_{F} (m) \leq ϵ} \leq min {m \in N : η_{m} \leq ω_{F^{*}}^{- 1} (ϵ; η)} .

Finally, by the definition of $F_{m}^{*}$ and $∥ (\dots, 0, u_{- m + 1 : 0}) - (\dots, 0, u_{- m + 1 : 0}^{'}) ∥_{η} \leq ∥ u_{- m + 1 : 0} - u_{- m + 1 : 0}^{'} ∥_{\infty}$ , we have

	$ω_{F_{m}^{*}} (δ)$	$= sup {\| F^{} (\dots, 0, u_{- m + 1 : 0}) - F^{} (\dots, 0, u_{- m + 1 : 0}^{'}) \| : ∥ u_{- m + 1 : 0} - u_{- m + 1 : 0}^{'} ∥_{\infty} \leq δ}$
		$≤ωF∗(δ;η).\qed$

In order to approximate the continuous causal time-invariant operator $F$ , we only need to approximate the functional $F^{*}$ , which can be approximated by the continuous function $F_{m}^{*}$ if $m$ is chosen sufficiently large. Hence, any approximation theory of continuous functions can be translated to an approximation result for continuous causal time-invariant operators. For instance, if we approximate $F_{m}^{*}$ by some function $h$ , then we can approximate $F$ by

y_{t} = h (u_{t - m + 1}, \dots, u_{t}), t \in Z .

The function $h$ uniquely determine a causal time-invariant operator $H$ such that $H (u)_{t} = h (u_{t - m + 1 : t})$ . Since $H$ has finite memory, $H$ is continuous if and only if $h$ is continuous by Proposition 2.3. When we approximate $F_{m}^{*}$ by polynomials, then $H$ is the Volterra series (Boyd and Chua, 1985). When $h$ is a neural network, then $H$

is a temporal convolution neural network, studied by

Hanson and Raginsky (2019).

In this paper, we focus on the approximation by echo state networks (ESN), which are special state space models of the form

{\begin{matrix} s_{t} = ϕ (W_{1} s_{t - 1} + W_{2} u_{t} + b), y_{t} = a \cdot s_{t}, \end{matrix}

(2.3)

where $s_{t}, a, b \in R^{n}$ , $W_{1} \in R^{n \times n}$ , $W_{2} \in R^{n \times d}$ and the activation function $ϕ : R \to R$ is applied element-wise.

Definition 2.4 (Existence of solutions and echo state property).

We say the system (2.3) has existence of solutions property if for any $u \in ([- 1, 1]^{d})^{Z_{-}}$ , there exist $s \in (R^{n})^{Z_{-}}$ such that $s_{t} = ϕ (W_{1} s_{t - 1} + W_{2} u_{t} + c)$ holds for each $t \in Z_{-}$ . If the solution $s$ is unique, we say the system has echo state property.

Grigoryeva and Ortega (2018, Theorem 3.1) gave sufficient conditions for the system (2.3) to have existence of solutions property and echo state property. In particular, they showed that, if $ϕ$ is a bounded continuous function, then the existence of solutions property holds. And if $ϕ$ is a bounded Lipschitz continuous function with Lipschitz constant $L_{ϕ} < \infty$ and $∥ W_{1} ∥ L_{ϕ} < 1$ , then the system (2.3) has echo state property, where $∥ W_{1} ∥$ is the operator norm of the matrix $W_{1}$ . As a sufficient condition to ensure the echo state property, the hypothesis $∥ W_{1} ∥ L_{ϕ} < 1$ has been been extensively studied in the ESN literature (Jaeger, 2001; Jaeger and Haas, 2004; Buehner and Young, 2006; Yildiz et al., 2012; Gandhi and Jaeger, 2013).

If the system (2.3) has existence of solutions property, the axiom of choice allows us to assign $s (u) \in (R^{n})^{Z_{-}}$ to each $u \in ([- 1, 1]^{d})^{Z_{-}}$ , and hence define a functional $h : ([- 1, 1]^{d})^{Z_{-}} \to R$ by $h (u) := a \cdot s (u)_{0} = y_{0}$ . Thus, we can assign a causal time-invariant operator $H : ([- 1, 1]^{d})^{Z} \to R^{Z}$ to the system such that $H^{*} = h$ . When the echo state property holds, this operator is unique. The operator $H$ is continuous if and only if the mapping $u \mapsto s (u)_{0}$ is a continuous function. In the next section, we study the universality of these operators.

3 Universal approximation

As mentioned in the introduction, the recent works of Grigoryeva and Ortega (2018); Gonon and Ortega (2021) showed that the echo state networks (2.3) are universal: Assume $ϕ$ is a bounded Lipschitz continuous function. Let $F : ([- 1, 1]^{d})^{Z} \to R^{Z}$ be a continuous causal time-invariant operator, then for any $ϵ > 0$ , for sufficiently large $k$ , there exists an ESN (2.3) such that the corresponding causal time-invariant operator $F_{E S N}$ satisfies

sup u \in ([- 1, 1]^{d})^{Z} ∥ F (u) - F_{E S N} (u) ∥_{\infty} \leq ϵ .

(3.1)

In this universal approximation theorem, the weights $W_{1}, W_{2}, a, b$ in the network (2.3) depend on the target operator $F$ . However, in practice, the parameters $W_{1}, W_{2}, b$

are drawn at random from certain given distribution and only the readout vector

a

are trained by linear regression using observed data related to the target operator

F

. Hence, this universal approximation theorem can not completely explain the empirical performance of echo state networks.

In this section, our goal is to show that, with randomly generated weights, echo state networks are universal: For any $ϵ > 0$ , for sufficiently large $k$ , with high probability on $W_{1}, W_{2}, b$ , which are drawn from certain distribution, there exists $a$ such that we can associate a causal time-invariant operator $F_{E S N}$ to the ESN (2.3) and the approximation bound (3.1

) holds. In the context of standard feed-forward neural networks, one can show that similar universal approximation theorem holds for random neural networks. This will be the building block of our main theorem of echo state networks.

3.1 Universality of random neural networks

It is well-known that feed-forward neural networks with one hidden layer are universal. We recall the universal approximation theorem proved by Leshno et al. (1993).

Theorem 3.1.

If $ϕ : R \to R$ is continuous and is not a polynomial, then for any compact set $X \subseteq R^{d}$ , any function $f \in C (X)$ and $ϵ > 0$ , there exists $a_{i}, b_{i} \in R$ and $w_{i} \in R^{d}$ , $i = 1, \dots, n$ , such that

sup x \in X ∣ ∣ ∣ ∣ f (x) - n \sum i = 1 a_{i} ϕ (w_{i} \cdot x + b_{i}) ∣ ∣ ∣ ∣ \leq ϵ .

(3.2)

In Theorem 3.1, the parameters $a_{i}, b_{i}, w_{i}$ depend on the target function $f$ . In order to take into account the fact that for ESN the inner weights $w_{i}, b_{i}$ are randomly chosen and only $a_{i}$ are trained from data, we consider the random neural networks whose weights $(w_{i}, b_{i})$

are drawn from some probability distribution

μ

. This motivates the following definition.

Definition 3.2.

Suppose $(w_{i}^{⊺}, b_{i})_{i \in N}$ is a sequence of i.i.d. random vectors drawn from probability distribution $μ$ defined on $R^{d + 1}$ . If for any compact set $X \subseteq R^{d}$ , any function $f \in C (X)$ and $ϵ, δ \in (0, 1)$ , there exists $n \in N$ such that, with probability at least $1 - δ$ , the inequality (3.2) holds for some $(a_{i})_{1 \leq i \leq n}$ , then we say the pair $(ϕ, μ)$ is universal.

Remark 3.3.

In the Definition 3.2, we only require the existence of neural networks that convergence to the target function in probability. This requirement is actually equivalent to almost sure convergence. To see this, convergence in probability implies that there exits a sub-sequence $n_{k}$ such that there exists $f_{n_{k}}$ , which is a linear combination of ${ϕ (w_{i} \cdot x + b_{i})}_{i = 1}^{n_{k}}$ , converging to the target function $f$ almost surely as $k \to \infty$ . Notice that, for any $n \geq n_{k}$ , $f_{n_{k}}$ is also a linear combination of ${ϕ (w_{i} \cdot x + b_{i})}_{i = 1}^{n}$ . Hence, the almost sure convergence holds for all $n$ .

The universality of random neural networks was widely studied in the context of extreme learning machine (Huang et al., 2006, 2006, 2012). In particular, Huang et al. (2006) used an incremental construction to establish the random universal approximation theorem in $L^{2}$ -norm for bounded non-constant piecewise continuous activation function. The recent work of Hart et al. (2020) considered the approximation in $C^{1}$ -norm and assumed that the activation function $ϕ \in C^{1} (R)$ satisfying $0 < \int_{R} | ϕ^{'} (x) | d x < \infty$ . They argued that, since there exits a neural network $g$ that approximates the target function $f$ by universality of neural networks, there will eventually be some randomly generated samples $(w_{i}^{⊺}, b_{i})$ that are close to the weights of $g$ and we can discard other samples by setting the corresponding $a_{i} = 0$

, hence the random universal approximation holds. It is possible to generalize their argument to the approximation of continuous functions. Nevertheless, we will give an alternative approach based on law of large numbers and show that

(ϕ, μ)

is universal under very weak conditions. Our analysis will need the following uniform law of large numbers.

Lemma 3.4 (Jennrich (1969), Theorem 2).

Let $w_{1}, \dots, w_{n}$ be i.i.d. samples from some probability distribution on $Ω$ . Suppose

[label=(0),parsep=0pt]
$X$ is compact;
$f (x, w)$ is continuous on $x$ for almost all $w \in Ω$ , and measurable on $w$ for each $x \in X$ ;
there exists $g (w)$ with $| f (x, w) | \leq g (w)$ for all $x \in X$ and $E_{w} [g (w)] < \infty$ .

Then $E_{w} [f (x, w)]$ is continuous on $x$ and

sup x \in X ∣ ∣ ∣ ∣ \frac{1}{n} n \sum i = 1 f (x, w_{i}) - E_{w} [f (x, w)] ∣ ∣ ∣ ∣ \to 0 almost % surely as n \to \infty .

Now, we give a sufficient condition for the pair $(ϕ, μ)$ to be universal. Our proof is a combination of the uniform law of large number (Lemma 3.4) and the universality of neural networks (Theorem 3.1).

Theorem 3.5.

Suppose the continuous function $ϕ : R \to R$ is not a polynomial and $| ϕ (x) | \leq C_{1} | x |^{α} + C_{2}$ for some $α, C_{1}, C_{2} \geq 0$ . If $μ$ is a probability distribution with full support on $R^{d + 1}$ such that $\int ∥ (w^{⊺}, b) ∥_{\infty}^{α} d μ (w^{⊺}, b) < \infty$ , then $(ϕ, μ)$ is universal.

Proof.

By Hahn-Banach theorem, for any compact set $X \subseteq R^{d}$ , the linear span of a function class $H \subseteq C (X)$ is dense in $C (X)$ if and only if

{ν \in M (X) : \int_{X} h (x) d ν (x) = 0, \forall h \in H} = {0},

where $M (X)$ is the dual space of $C (X)$ , that is, the space of all signed Radon measures with finite total variation (Folland, 1999).

We consider the linear space

H_{μ} := {h (x) = \int_{R^{d + 1}} g (w, b) ϕ (w \cdot x + b) d μ (w^{⊺}, b) : g \in L^{\infty} (μ)} .

Since $| g (w, b) ϕ (w \cdot x + b) | \leq ∥ g ∥_{\infty} (C_{1} | w \cdot x + b |^{α} + C_{2}) \leq C_{X} ∥ g ∥_{\infty} ∥ (w^{⊺}, b) ∥_{\infty}^{α} + C_{2} ∥ g ∥_{\infty}$ , by assumption and Lemma 3.4, any $h \in H_{μ}$ is continuous and hence $H_{μ} \subseteq C (X)$ . Suppose $ν \in M (X)$ satisfies $\int_{X} h (x) d ν (x) = 0$ for all $h \in H_{μ}$ . Then, by Fubini’s theorem,

	$0 =$	$\int_{X} \int_{R^{d + 1}} g (w, b) ϕ (w \cdot x + b) d μ (w^{⊺}, b) d ν (x)$
	$=$	$\int_{R^{d + 1}} g (w, b) \int_{X} ϕ (w \cdot x + b) d ν (x) d μ (w^{⊺}, b),$

for all function $g \in L^{\infty} (μ)$ . Therefore,

\int_{X} ϕ (w \cdot x + b) d ν (x) = 0 % almost surely .

Since this function is continuous and $μ$ has full support, the equality holds for all $(w^{⊺}, b) \in R^{d + 1}$ . By Theorem 3.1, the linear span of ${ϕ (w \cdot x + b) : (w^{⊺}, b) \in R^{d + 1}}$ is dense in $C (X)$ , which implies $ν = 0$ . We conclude that $H_{μ}$ is dense in $C (X)$ . In other words, for any $f \in C (X)$ and $ϵ > 0$ , there exists $h \in H_{μ}$ such that $| f (x) - h (x) | \leq ϵ / 2$ for all $x \in X$ .

By Lemma 3.4, for any $δ > 0$ , there exists $n \in N$ such that, with probability at least $1 - δ$ on the samples $(w_{i}^{⊺}, b_{i})$ from $μ$ ,

sup x \in X ∣ ∣ ∣ ∣ \frac{1}{n} n \sum i = 1 g (w_{i}, b_{i}) ϕ (w_{i} \cdot x_{i} + b_{i}) - h (x) ∣ ∣ ∣ ∣ \leq \frac{ϵ}{2} .

By triangle inequality, we conclude that (3.2) holds with $a_{i} = \frac{1}{n} g (w_{i}, b_{i})$ and the pair $(ϕ, μ)$ is universal. ∎

In practice, it is more convenient to sample each weight independently from certain distribution $μ$ on $R$ , so that $(w^{⊺}, b)$ is sample from $μ^{d + 1}$ , the $d + 1$ products of $μ$ . The next corollary is a direct application of Lemma 3.5 to this situation.

Corollary 3.6.

Suppose the continuous function $ϕ : R \to R$ is not a polynomial and $| ϕ (x) | \leq C_{1} | x |^{α} + C_{2}$ for some $α, C_{1}, C_{2} \geq 0$ . If $μ$ is a probability distribution with full support on $R$ such that $\int | x |^{α} d μ (x) < \infty$ , then for any $d \in N$ , the pair $(ϕ, μ^{d + 1})$ is universal.

Proof.

When $0 \leq α \leq 1$ ,

	$\int ∥ (w^{⊺}, b) ∥_{\infty}^{α} d μ^{d + 1} (w^{⊺}, b)$	$\leq \int ∥ (w^{⊺}, b) ∥_{1}^{α} d μ^{d + 1} (w^{⊺}, b)$
		$\leq \int \| b \|^{α} + d \sum i = 1 \| w_{i} \|^{α} d μ^{d + 1} (w^{⊺}, b) < \infty .$

When $α > 1$ , since every norm of $R^{d + 1}$ are equivalent,

	$\int ∥ (w^{⊺}, b) ∥_{\infty}^{α} d μ^{d + 1} (w^{⊺}, b)$	$\leq C \int ∥ (w^{⊺}, b) ∥_{α}^{α} d μ^{d + 1} (w^{⊺}, b)$
		$= C \int \| b \|^{α} + d \sum i = 1 \| w_{i} \|^{α} d μ^{d + 1} (w^{⊺}, b) < \infty .$

In any cases, the pair $(ϕ, μ^{d + 1})$ satisfies the condition in Lemma 3.5, hence it is universal. ∎

So far, we have assumed that $μ$ has full support. When the activation $ϕ (x) = σ (x) = max {x, 0}$ is the ReLU function, this assumption can be weaken due to the absolute homogeneity of ReLU.

Corollary 3.7.

For the ReLU function $σ$ , if $μ$ is a probability distribution on $R$ whose support contains the interval $[- r, r]$ for some $r > 0$ , then $(σ, μ^{d + 1})$ is universal for any $d \in N$ .

Proof.

We consider the continuous mapping $T : R^{d + 1} \to [- r, r]^{d + 1}$ defined by

T (w^{⊺}, b) = {\begin{matrix} (w^{⊺}, b), & ∥ (w^{⊺}, b) ∥_{\infty} \leq r \frac{r}{∥ (w^{⊺}, b) ∥_{\infty}} (w^{⊺}, b) & ∥ (w^{⊺}, b) ∥_{\infty} > r . \end{matrix}

Let $γ = T_{#} μ^{d + 1}$ be the push-forward measure of $μ^{d + 1}$ under $T$ defined by $γ (S) = μ^{d + 1} (T^{- 1} (S))$ for any measurable set $S \subseteq [- r, r]^{d + 1}$ . Then, the support of $γ$ is $[- r, r]^{d + 1}$ by assumption.

We firstly show that $(σ, γ)$ is universal. As in the proof of Theorem 3.5, if $ν \in M (X)$ satisfies $\int_{X} h (x) d ν (x) = 0$ for all $h \in H_{γ}$ , then by Fubini’s theorem,

\int_{X} σ (~ w \cdot x + ~ b) d ν (x) = 0

holds for all $({~ w}^{⊺}, ~ b) \in [- r, r]^{d + 1}$ . By the absolute homogeneity of $σ$ , this equation actually holds for all $({~ w}^{⊺}, ~ b) \in R^{d + 1}$ . The argument in the proof of Theorem 3.5 implies that $(σ, γ)$ is universal.

Observe that any sample $(w_{i}^{⊺}, b_{i})$ from $μ^{d + 1}$ corresponding to a sample $({~ w}_{i}^{⊺}, {~ b}_{i}) = T (w_{i}^{⊺}, b_{i})$ from $γ = T_{#} μ^{d + 1}$ , and

n \sum i = 1 a_{i} ϕ (w_{i} \cdot x + b_{i}) = n \sum i = 1 \frac{a_{i}}{c_{i}} ϕ ({~ w}_{i} \cdot x + {~ b}_{i}),

where $c_{i} = 1$ if $∥ (w_{i}^{⊺}, b_{i}) ∥_{\infty} \leq r$ and $c_{i} = ∥ (w_{i}^{⊺}, b_{i}) ∥_{\infty} / r$ otherwise. We conclude that $(σ, μ^{d + 1})$ is universal. ∎

3.2 Universality of echo state networks

In this section, we will state and prove the random universal approximation theorem for echo state networks. Our analysis is based on the uniform law of large numbers and the universality of random feed-forward neural networks. For simplicity, we will assume that all internal weights in the network have the same distribution $μ$ from now on. We make the following assumption on the activation function $ϕ : R \to R$ and the distribution $μ$ .

Assumption 3.8.

For any $m \in N$ , the pair $(ϕ, μ^{m + 1})$ is universal and there exists a measurable mapping $φ_{μ, m} : R^{m + 1} \to R^{m}$ such that

x = E_{(w^{⊺}, b) \sim μ^{m + 1}} [φ_{μ, m} (w, b) ϕ (w \cdot x + b)], x \in [- 2, 2]^{m},

(3.3)

and $∥ φ_{μ, m} (w, b) ϕ (w \cdot x + b) ∥_{\infty} \leq g_{m} (w, b)$ for some $g_{m}$ satisfying $E [g_{m} (w, b)] < \infty$ .

Corollary 3.6 gives sufficient condition for the pair $(ϕ, μ^{m + 1})$ to be universal. However, in Assumption 3.8, we also require the existence of function $φ_{μ, m}$ that satisfies equation (3.3), which may be difficult to check for general activation functions. Nevertheless, we will show that the assumption holds true when the activation function is ReLU and $μ$ is symmetric (see Corollary 3.11).

By Lemma 3.4, Assumption 3.8 ensures that if we have $n$ i.i.d. samples $(w_{1}^{⊺}, b_{1}), \dots, (w_{n}^{⊺}, b_{n})$ from $μ^{m + 1}$ , we can approximately reconstruct the input by

x \approx \frac{1}{n} n \sum i = 1 φ_{μ, m} (w_{i}, b_{i}) ϕ (w_{i} \cdot x + b_{i}) .

In other words, the features $ϕ (w_{i} \cdot x + b_{i})$ contain enough information of the input and we can approximately recover it using $φ_{μ, m} (w_{i}, b_{i})$ as coefficients. For echo state networks, this assumption guarantees that the hidden state does not lose too much information about the history of the input, and hence we can view it as a “reservoir” and approximate the desired function at any time steps.

Theorem 3.9.

Suppose $ϕ : R \to R$ and the distribution $μ$ satisfy Assumption 3.8. For $n, m, d \in N$ , let $(w_{1}^{⊺}, b_{1}), \dots, (w_{n}^{⊺}, b_{n})$ be $n$ i.i.d. samples from $μ^{m d + 1}$ , and define the ESN

{\begin{matrix} s_{t} = ϕ \end{matrix}

Universality and approximation bounds for echo state networks with random weights

Approximation Bounds for Random Neural Networks and Reservoir Systems

Expressive power of binary and ternary neural networks

Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems

Neural networks with superexpressive activations and integer weights

Universal Approximation Power of Deep Neural Networks via Nonlinear Control Theory

Sharp asymptotics on the compression of two-layer neural networks

Neural tangent kernels, transportation mappings, and universal approximation

1 Introduction

1.1 Notations

2 Continuous causal time-invariant operators

Definition 2.1 (Approximately finite memory).

Definition 2.2 (Modulus of continuity).

Proposition 2.3.

Proof.

Definition 2.4 (Existence of solutions and echo state property).

3 Universal approximation

3.1 Universality of random neural networks

Theorem 3.1.

Definition 3.2.

Remark 3.3.

Lemma 3.4 (Jennrich (1969), Theorem 2).

Theorem 3.5.

Proof.

Corollary 3.6.

Proof.

Corollary 3.7.

Proof.

3.2 Universality of echo state networks

Assumption 3.8.

Theorem 3.9.

Universality and approximation bounds for echo state networks with random weights

Related Research

Approximation Bounds for Random Neural Networks and Reservoir Systems

Expressive power of binary and ternary neural networks

Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems

Neural networks with superexpressive activations and integer weights

Universal Approximation Power of Deep Neural Networks via Nonlinear Control Theory

Sharp asymptotics on the compression of two-layer neural networks

Neural tangent kernels, transportation mappings, and universal approximation

1 Introduction

1.1 Notations

2 Continuous causal time-invariant operators

Definition 2.1 (Approximately finite memory).

Definition 2.2 (Modulus of continuity).

Proposition 2.3.

Proof.

Definition 2.4 (Existence of solutions and echo state property).

3 Universal approximation

3.1 Universality of random neural networks

Theorem 3.1.

Definition 3.2.

Remark 3.3.

Lemma 3.4 (Jennrich (1969), Theorem 2).

Theorem 3.5.

Proof.

Corollary 3.6.

Proof.

Corollary 3.7.

Proof.

3.2 Universality of echo state networks

Assumption 3.8.

Theorem 3.9.