Some negative results for Neural Networks

1. Introduction

The standard model of the single hidden layer feedforward neural networks leads to the problem of approximation of arbitrary functions $f : R^{d} \to R$ by elements of the set

Σ_{n}^{σ, d} = {n \sum k = 1 c_{k} σ (w^{k} \cdot x - b_{k}) : w^{k} \in R^{d}, c_{k}, b_{k} \in R},

where $σ \in C (R)$ is the given activation function of the net and $w^{k} \cdot x = \sum_{i = 1}^{d} w_{i k} x_{i}$

is the dot product of the vectors

w^{k} = (w_{1 k}, \dots, w_{d k})

and

x = (x_{1}, \dots, x_{d})

It is well known that $⋃_{n = 1}^{\infty} Σ_{n}^{σ, d}$ is dense in $C (R^{d})$ (for the topology of uniform convergence on compact subsets of $R^{d}$ ) if and only if $σ$ is not a polynomial (see, e.g. [9, Theorem 1] and [8] for a different density result). In particular, this means that feedforward networks with a nonpolynomial activation function can approximate any continuous function and, thus, are good for any learning objective in the sense that, given a precision $ε > 0$ there exists $n \in N$ with the property that the associated feedforward neural network with one hidden layer and $n$

units can be (in principle) trained to approximate the associated learning function

f (x)

with uniform accuracy smaller than

ε

. In other words, we know that for any nonpolynomial activation function

σ \in C (R)

and for any compact set

K \subset R^{d}

and any

f \in C (K)

E (f, Σ_{n}^{σ, d})_{C (K)} = inf g \in Σ_{n}^{σ, d} ∥ f - g ∥_{C (K)} \leq ε

for $n$ large enough. These are of course good news for people working with neural networks. On the other hand, several examples of high-oscillating functions such as those studied at [14], [16] have shown that sometimes it is necessary to increase dramatically the number of units (and layers) of a neural network if one wants to approximate certain functions. But these examples do not contain, by themselves, any information about the decay of best approximation errors when the size of the neural network goes to infinity, so that one considers the full sequence of best approximation errors. In this paper we demonstrate a negative result which, in philosophical terms, claims the existence of learning functions $f (x)$ which are as difficult to approximate with neural networks with one hidden layer as one may want and we also demonstrate an analogous result for neural networks with an arbitrary number of layers for some special types of activation functions $σ$ . Concretely, in section 2 we demonstrate that for any non polynomial activation function $σ \in C (R)$ , for any natural number $d \geq 2$ and for any compact set $K \subseteq R^{d}$ with nonempty interior and any given decreasing sequence of positive real numbers ${ε_{n}}$ which converges to zero, there exists a continuous function $f \in C (K)$ such that

E (f, Σ_{n}^{σ, d})_{C (K)} \geq ε_{n}, for all n = 1, 2, \dots .

We also demonstrate the same type of result for the norms $L^{q} (K)$ , $2 \leq q < \infty$ . The proofs of these theorems are based on the combination of a general negative result in approximation theory demonstrated by Almira and Oikhberg in 2012 [1] (see also [2]

) and the estimations of the deviations of Sobolev classes by ridge functions proved by Gordon, Maiorov, Meyer and Reisner in 2002

[5]. It is important to stand up the fact that our result requires the use of functions of several variables and can’t be applied to univariate functions. This is natural not only because of the method of proof we use, which is based on the statement of a negative result for approximation by ridge functions that holds true only for

d \geq 2

, but also because quite recently Guliyev and Ismailov [6, Theorems 4.1 and 4.2] have demonstrated that for the case

d = 1

a general negative result for approximation by single hidden layer feedforward neural networks is impossible. In particular they have explicitly constructed an infinitely differentiable sigmoidal activation function

σ

such that

Σ_{2}^{σ, 1}

is a dense subset of

C (R)

with the topology of uniform convergence on compact subsets of

R

On the other hand, as a consequence of Kolmogorov’s Superposition Theorem [4, Chapter 17, Theorem 1.1], a general negative result for approximation by several hidden layer feedforward neural networks is impossible not only for univariate ( $d = 1$ ) but also for arbitrary multivariate ( $d \geq 2$ ) functions (see [10] for a proof of this claim and [7, 11, 12] for other interesting related results). Nevertheless, in section 3 we demonstrate several negative results for specific choices of activation functions $σ$ and for arbitrary $d \geq 1$ . Concretely, we prove that if $σ$ is either a rational (nonpolynomial) function or a linear spline with finitely many pieces (which is nonpolynomial too), then for any pair of sequences of natural numbers $0 = r_{0} < r_{1} < r_{2} < \dots < r_{k} < \dots$ , $0 = n_{0} < n_{1} \leq n_{2} \leq \dots \leq n_{k} \leq \dots$ and any non-increasing sequence ${ε_{n}}$ which converges to $0$ , and for appropriate compact subsets $K$ of $R^{d}$ , there exists a function $f \in C (K)$ such that

E (f, τ_{r_{k}, n_{k}}^{σ, d})_{C (K)} \geq ε_{k} for all k = 1, 2, \dots,

where $τ_{r, n}^{σ, d}$ denotes the set of functions defined on $R^{d}$ by a neural network with activation function $σ$ and at most $r$ layers and $n$ units in each layer. It is important to stand up the fact that the results of this section apply for all values of $d \geq 1$ and, moreover, they also apply to neural networks with activation functions

σ (t) = ReLU (t) = {\begin{matrix} 0 & t < 0 t & t \geq 0 \end{matrix}

and

σ (t) = Hard Tanh (t) = ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} - 1 & t < - 1 t & - 1 \leq t \leq 1 1 & t > 1 \end{matrix},

which are two of the most used activation functions for people working with neural networks.

2. A general negative result for feedforward neural networks with one hidden layer

Let us start by introducing some notation and terminology from Approximation Theory. Given $(X, ∥ \cdot ∥)$ a Banach space, we say that $(X, {A_{n}})$ is an approximation scheme (or that ${A_{n}}$ is an approximation scheme in $X$ ) if ${A_{n}}$ satisfies the following properties:

$A_{0} = {0} \subset A_{1} \subset \dots \subset A_{n} \subset A_{n + 1} \subset \dots$ is a nested sequence of subsets of $X$ (with strict inclusions)
There exists a map $J : N \to N$ such that $J (n) \geq n$ and $A_{n} + A_{n} \subseteq A_{J (n)}$ for all $n \in N$
$λ A_{n} \subseteq A_{n}$ for all $n \in N$ and all scalars $λ$ .
$⋃_{n \in N} A_{n}$ is dense in $X$ .

It is known that for any activation function $σ \in C (R)$ and any infinite compact set $K \subseteq R^{d}$ , $(C (K), {Σ_{n}^{σ, d}})$ is an approximation scheme, with jump function $J (n) = 2 n$ , as soon as $σ$ is not a polynomial, since the sets ${Σ_{n}^{σ, d}}$ always satisfy $(A 1)$ , $(A 2)$ and $(A 3)$ and, given an infinite compact set $K \subseteq R^{d}$ , they also satisfy $(A 4)$ if (and only if) $σ$ is not a polynomial (see [9]). In fact, if $σ$ is not a polynomial, then $(L^{p} (K), {Σ_{n}^{σ, d}})$ is an approximation scheme for every compact set $K \subset R^{d}$ of positive measure and all $p \geq 1$ .

Another important example of approximation scheme, again with jump function $J (n) = 2 n$ , is given by $(X, {R_{n}^{d}})$ , where $d \geq 2$ , $X = C (K)$ or $X = L^{p} (K)$ (with $p \geq 1$ ) for some compact set $K \subseteq R^{d}$ with positive Lebesgue measure, $R_{0}^{d} = {0}$ and, for $n \geq 1$ ,

R_{n}^{d} = {n \sum i = 1 g_{i} (a^{i} \cdot x) : a^{i} \in S^{d - 1}, g_{i} \in C (R), i = 1, \dots, n},

where $S^{d - 1} = {x \in R^{d} : ∥ x ∥_{2}^{2} = \sum_{i = 1}^{d} | x_{i} |^{2} = 1}$ . The elements of $R_{n}^{d}$ are called ridge functions (in $d$ variables). The density of ridge functions in these spaces is a well known fact (see [13]).

We prove, for the sake of completeness, the following result:

Proposition 1.

Σ_{n}^{σ, d} \subset R_{n}^{d}

Proof. Let $ϕ (x) = \sum_{k = 1}^{n} c_{k} σ (w^{k} \cdot x - b_{k})$ be any element of $Σ_{n}^{σ, d}$ and let us set $g_{k} (t) = c_{k} σ (∥ w^{k} ∥ t - b_{k})$ and $a^{k} = \frac{1}{∥ w^{k} ∥} w^{k}$ . Then $g_{k} \in C (R)$ and $a^{k} \in S^{d - 1}$ for $k = 1, \dots, d$ . Moreover,

d \sum k = 1 g_{k} (a^{k} \cdot x) = d \sum k = 1 c_{k} σ (∥ w^{k} ∥ a^{k} \cdot x - b_{k}) = d \sum k = 1 c_{k} σ (w^{k} \cdot x - b_{k}) = ϕ (x),

which means that $ϕ \in R_{n}^{d}$ . $□$

In [1, Theorems 2.2 and 3.4] (see also [2, Theorem 1.1]) the following result was proved

Theorem 2 (Almira and Oikhberg, 2012).

Given ${A_{n}}$ an approximation scheme on the Banach space $X$ , the following are equivalent claims:

For every non-increasing sequence ${ε_{n}} \to 0$ there exists an element $x \in X$ such that

$E (x, A_{n}) = inf a_{n} \in A_{n} ∥ x - a_{n} ∥_{X} \geq ε_{n} for % all n \in N$
There exists a constant $c > 0$ and an infinite set $N_{0} \subseteq N$ such that, for all $n \in N_{0}$ there exists $x_{n} \in X ∖_{n}$ such that

$E (x_{n}, A_{n}) \leq c E (x_{n}, A_{J (n)})$

Remark 3.

If $(X, {A_{n}})$ satisfies $(a)$ or $(b)$ of Theorem 2 we say that the approximation scheme satisfies Shapiro’s theorem.

Let us state the main result of this section:

Theorem 4.

Let $σ \in C (R)$ be any nonpolynomial activation function, let $d \geq 2$ be a natural number and let $K \subset R^{d}$ be a compact set with nonempty interior. For any non-increasing sequence of real numbers ${ε_{n}}$ such that ${lim}_{n \to \infty} ε_{n} = 0$ , there exist a learning function $f \in C (K)$ such that

E (f, Σ_{n}^{σ, d})_{C (K)} \geq E (f, R_{n}^{d})_{C (K)} \geq ε_{n} for all n = 1, 2, \dots .

Proof. It is enough to demonstrate that the approximation scheme $(C (K), {R_{n}^{d}})$ satisfies condition $(b)$ of Theorem 2, since Proposition 1 implies that

E (f, Σ_{n}^{σ, d})_{C (K)} \geq E (f, R_{n}^{d})_{C (K)}

for all $n$ . In order to do this, we need to use the following result (see [5, Theorem 1.1])

Theorem 5 (Gordon, Maiorov, Meyer and Reisner, 2002).

Let $d \geq 2$ and let $r > 0$ and $2 \leq q \leq p \leq \infty$ . Then the asymptotic relation

dist (B (W_{p}^{r, d}), R_{n}^{d}, L_{q}) ≍ n^{\frac{- r}{d - 1}}

holds.

Here, $B (W_{p}^{r, d})$ denotes the unit ball in the Sobolev-Slobodezkii class of functions $W_{p}^{r, d} (B_{d})$ defined on the unit ball of $R^{d}$ (but the result holds true with the same asymptotics for all balls of $R^{d}$ ) and

dist (B (W_{p}^{r, d}), R_{n}^{d}, L^{q}) = sup f \in B (W_{p}^{r, d}) E (f, R_{n}^{d})_{L^{q} (B_{d})} .

We apply Theorem 5 to functions defined on a ball of $R^{d}$ of radius $t > 0$ , $B_{d} (t) \subset K$ . Thus, given $r > 0$ and $2 \leq q \leq p \leq \infty$ , there exists two positive constants $c_{0}, c_{1} > 0$ such that:

For any $f \in B (W_{p}^{r, d})$ ,

$E (f, R_{n}^{d})_{L^{q} (B_{d} (t))} \leq c_{1} n^{\frac{- r}{d - 1}} .$
For any $m \in N$ there exists $f_{m} \in B (W_{p}^{r, d})$ such that

$E (f_{m}, R_{m}^{d})_{L^{q} (B_{d} (t))} \geq c_{0} m^{\frac{- r}{d - 1}} .$

Combining these inequalities for $m = 2 n$ , $n \in N$ , we have that

	$E (f_{2 n}, R_{2 n}^{d})_{L^{q} (B_{d} (t))}$	$\geq$	$c_{0} (2 n)^{\frac{- r}{d - 1}} = c_{0} 2^{\frac{- r}{d - 1}} n^{\frac{- r}{d - 1}}$
		$\geq$	$2^{\frac{- r}{d - 1}} \frac{c_{0}}{c_{1}} E (f_{2 n}, R_{n}^{d})_{L^{q} (B_{d} (t))}$

Hence

E (f_{2 n}, R_{n}^{d})_{L^{q} (B_{d} (t))} \leq 2^{\frac{r}{d - 1}} \frac{c_{1}}{c_{0}} E (f_{2 n}, R_{2 n}^{d})_{L^{q} (B_{d} (t))} for all n \in N

and $(L^{q} (B_{d} (t)), {R_{n}^{d}})$ satisfies condition $(b)$ of Theorem 2, which implies that, for any non-increasing sequence of positive numbers ${ε_{n}} \to 0$ there exists a function $f \in L^{q} (B_{d} (t))$ such that

Taking $¯ ¯ ¯ f \in L^{q} (K)$ any extension of $f$ to the compact set $K$ , we also have that

E (¯ ¯ ¯ f, R_{n}^{d})_{L^{q} (K)} \geq ε_{n} for all n \in N .

Moreover, in the case $p = q = \infty$ , we can use Sobolev’s embedding theorem to guarantee that, for $r$ big enough, the Sobolev class $W_{\infty}^{r, d}$ is formed by continuous functions, which implies that the inequalities

E (f_{2 n}, R_{n}^{d})_{L^{\infty} (B_{d} (t))} \leq 2^{\frac{r}{d - 1}} \frac{c_{1}}{c_{0}} E (f_{2 n}, R_{2 n}^{d})_{L^{\infty} (B_{d} (t))} for all n \in N

can be stated for continuous functions:

E (f_{2 n}, R_{n}^{d})_{C (B_{d} (t))} \leq 2^{\frac{r}{d - 1}} \frac{c_{1}}{c_{0}} E (f_{2 n}, R_{2 n}^{d})_{C (B_{d} (t))} for all n \in N

and we can apply Theorem 2 to the approximation scheme $(C (B_{d} (t)), {R_{n}^{d}})$ and, consequently, to the approximation scheme $(C (K), {R_{n}^{d}})$ . This ends the proof. $□$

3. Negative results for feedforward neural networks with several hidden layers

In [10, Theorem 4 and comments below it] it was proved that, for a proper choice of the activation function $σ$ , which may be chosen real analytic and of sigmoidal type, a feedforward neural network with two layers and a fixed finite number of units in each layer is enough for uniform approximation of arbitrary continuous functions on any compact set $K \subseteq R^{d}$ . Concretely, the result was demonstrated for neural networks with $3 d$ units in the first layer and $6 d + 3$ units in the second layer. Moreover, reducing the restrictions on $σ$ , the same result can be obtained for a neural network of two layers with $d$ units in the first layer and $2 d + 1$ units in the second layer. Finally, Guliyev and Ismailov have recently shown, for $d \geq 2$ , an algorithmically constructed two hidden layers feedforward neural network with fixed weights which has a total of $3 d + 2$ units and uniformly approximates arbitrary continuous functions on compact subsets of $R^{d}$ (see [7]).

In this section we prove that, for certain natural choices of $σ$ , a negative result holds true for neural networks with any number of layers and units in each layer. Concretely, we demonstrate a negative result for multivariate uniform approximation by rational functions and spline functions with free partitions and, as a consequence, we get a negative result for approximation by feedforward neural networks with arbitrary number of layers when the activation function $σ$ is either a rational (non polynomial) function or a linear spline with finitely many pieces (e.g., the well known ReLU and Hard Tanh functions are linear spline functions with two and three pieces, respectively). Note that the use of rational and/or spline approximation tools in connection with the study of these classical neural networks is a natural one and has been faced by several other authors (see, e.g. [14, 15, 16, 17, 18]).

In [1, Sections 6.3 and 6.4] the one dimensional case of rational and free partitions spline approximation were studied, so that we can state the following two results:

Theorem 6.

Let $R_{n}^{1} (I)$ denote the set of rational functions $r (x) = p (x) / q (x)$ with $max {deg (p), deg (q)} \leq n$ and such that $q (x)$ does not vanish on $I$ , and assume that $I = [a, b] \subset R$ with $a < b$ . Let $1 \leq n_{1} < n_{2} < \dots$ be an increasing sequence of natural numbers. Let $A_{0} = {0}$ and $A_{i} = R_{n_{i}}^{1} (I)$ , $i > 0$ . Then $(C (I), {A_{i}})$ is an approximation scheme which satisfies Shapiro’s theorem.

Proof. The result follows from [1, Item $(1)$ from Theorem 6.9] for the case $n_{i} = i$ , $i = 1, 2, \dots$ . Let us now assume that $1 \leq n_{1} < n_{2} < \dots$ is any other sequence. Given ${ε_{n}}$ a non-increasing sequence which converges to $0$ , we introduce the new sequence

ϵ_{n} = ε_{k}, when n_{k - 1} \leq n < n_{k} .

Then ${ϵ_{n}}$ is non-increasing and ${lim}_{n \to \infty} ϵ_{n} = 0$ . Thus, there exists a continuous function $f \in C (I)$ such that $E (f, R_{n}^{1} (I))_{C (I)} \geq ϵ_{n}$ for all $n$ . In particular,

E (f, A_{i}) = E (f, R_{n_{i}}^{1} (I)) \geq ϵ_{n_{i}} = ε_{i} for all i \in N .

This ends the proof. $□$

Theorem 7.

Let $S_{n, r} (I)$ denote the set of polynomial splines of degree $\leq n$ with $r$ free knots in the interval $I = [a, b]$ . Let $0 \leq r_{1} < r_{2} < \dots$ and $1 \leq n_{1} < n_{2} < \dots$ be a pair of increasing sequences of natural numbers. Let $A_{i} = S_{n_{i}, r_{i}} \cap C (I)$ and $B_{i} = S_{n_{i}, r_{i}}$ and let $1 \leq p < \infty$ . Then $(C (I), {A_{i}})$ and $(L_{p} (I), {B_{i}})$ are approximation schemes which satisfy Shapiro’s Theorem.

Proof. See [1, Theorem 6.12] $□$

Let us now study the multivariate case. For the case of rational approximation, we consider, for compact sets $K \subset R^{d}$ , the approximation schemes $(C (K), {R_{n}^{d} (K)})$ , where

R_{n}^{d} (K) = {\frac{p (x_{1}, \dots, x_{d})}{q (x_{1}, \dots, x_{d})} : p, q are polynomials, max {deg (p), deg (q)} \leq n} \cap C (K) .

Theorem 8.

For any convex compact set $K \subset R^{d}$ and any increasing sequence of natural numbers $1 \leq n_{1} < n_{2} < \dots n_{k} < \dots$ , the approximation scheme $(C (K), {R_{n_{k}}^{d} (K)})$ satisfies Shapiro’s theorem.

Proof. Let $ρ = diam (K) = {max}_{x, y \in K} ∥ x - y ∥$ . We may assume, with no loss of generality, that $W = K \cap R \times {0} \times \dots \times {0} = [a, a + ρ] \times {0} \times \dots \times {0}$ for certain $a \in R$ , since rotations and translations of the space preserve the set of rational functions of any given order. Thus, the functions $ϕ (s) = r (s, 0, \dots, 0)$ with $r \in R_{n}^{d} (K)$ satisfy $ϕ \in R_{n}^{1} ([a, a + ρ])$ . Moreover, the convexity of $K$ and the fact that $ρ = diam (K)$ imply that every function $ϕ \in R_{n}^{1} ([a, a + ρ])$ extends to a function $r \in R_{n}^{d} (K)$ just taking $r (x_{1}, x_{2}, \dots, x_{d}) = ϕ (x_{1})$ .

Hence, for any $f \in C (K)$ we have that

$E (f, R_{n}^{d} (K))_{C (K)}$	$=$	$inf r \in R_{n}^{d} (K) sup x \in K \| f (x) - r (x) \|$
	$\geq$	$inf r \in R_{n}^{d} (K) sup x \in [a, a + ρ] \times {0} \times \dots \times {0} \| f (x) - r (x) \|$
	$=$	$inf ϕ \in R_{n}^{1} ([a, a + ρ]) sup s \in [a, a + ρ] \| f (s, 0, \dots, 0) - ϕ (s) \|$
	$=$	$E (g, R_{n}^{1} ([a, a + ρ]))_{C [a, a + ρ]},$

where $g (s) = f (s, 0, \dots, 0)$ is a continuous function on the interval $[a, a + ρ]$ . Now, given any function $g \in C [a, a + ρ]$ there exists a continuous function $¯ ¯ ¯ g \in C (K)$ such that $g (s) = ¯ ¯ ¯ g (s, 0, \dots, 0)$ for all $s \in [a, a + ρ]$ (e.g., we may take $¯ ¯ ¯ g (x_{1}, x_{2}, \dots, x_{d}) = g (x_{1})$ ). Hence $E (¯ ¯ ¯ g, R_{n}^{d} (K))_{C (K)} \geq E (g, R_{n}^{1} ([a, a + ρ]))_{C [a, a + ρ]}$ for all $n$ . The proof ends applying Theorem 6 to the space $C [a, a + ρ]$ . $□$

Given a polyhedron $K \subseteq R^{d}$ , we say that $Γ = {Δ_{k}}_{k = 1}^{r}$ is a triangulation of $K$ if

Every set $Δ_{k}$ is a $d$ -simplex. This means that $Δ_{k}$ is the convex hull of a set of $d + 1$ affinely independent points of $R^{d}$ .
$⋃_{k = 1}^{r} Δ_{k} = K$ and, for $i \neq j$ , $Δ_{i} \cap Δ_{j}$ is either empty or an $h$ -simplex for some $h < d$ (thus, it is the convex hull of a set of $h + 1$ affinely independent points of $R^{d}$ ).

If the polyhedron admits a triangulation $Γ$ we say that it is triangularizable. For example, all convex polyhedron is triangularizable. We consider, for any trianguralizable polyhedron $K \subseteq R^{d}$ , the approximation schemes $(C (K), {S_{n, n}^{d} (K)}_{k = 0}^{\infty})$ , where $S_{r, n}^{d} (K)$ denotes the set of spline functions defined on a partition of $K$ with at most $r$ simplices and $S_{0, 0}^{d} (K) := {0}$ . Concretely, $ϕ$ is an element of $S_{r, n}^{d} (K)$ if and only if it is a continuous function on $K$ and satisfies the identity

ϕ (x) = \sum Δ \in Γ p_{Δ} (x) χ_{Δ} (x) for all x \in K,

for some triangulation $Γ$ of $K$ with $# Γ \leq r$ , where $p_{Δ}$ denotes, for each $Δ \in Γ$ , a polynomial of total degree $deg (p) \leq n$ .

We need, to prove Shapiro’s theorem for approximation with spline functions on free partitions, the following technical result on triangulations:

Proposition 9.

Given $d \geq 1$ , there exists a function $h : N \times N \to N$ such that, if $K \subset R^{d}$ is a convex polyhedron and $Γ_{1}, Γ_{2}$ are two triangulations of $K$ with cardinalities $# Γ_{1} = n$ , $# Γ_{2} = m$ , then there exists a triangulation $Γ$ of $K$ which is a refinement of $Γ_{1}$ and $Γ_{2}$ and satisfies that $# Γ \leq h (n, m)$ .

Proof. Given $Γ_{1}, Γ_{2}$ two triangulations of $K$ with cardinalities $# Γ_{1} = n$ , $# Γ_{2} = m$ , let us consider the following set:

Θ = {Δ_{1} \cap Δ_{2} : Δ_{i} is a simplex of Γ_{i}, i = 1, 2} .

Obviously, $Θ$ is a polyhedral subdivision of both $Γ_{1}, Γ_{2}$ but in many cases is not a triangulation of $K$ . Moreover, the number $N$ of vertices of $Θ$ will, in general, be bigger than $n$ and $m$ . Let us find an upper bound of $N$ . Assume that $v$ is a new vertex of $Θ$ (i.e., a vertex of $Θ$ which is not a vertex of $Γ_{1} \cup Γ_{2}$ ) and let us consider the smaller simplices $δ_{i} \in Γ_{i}$ ( $i = 1, 2$ ) which contain $v$ in its relative interior (that is, in the interior of the simplex with its relative topology). Then ${v} = δ_{1} \cap δ_{2}$ . Hence the number of new vertices can’t be bigger than the product of the number of simplices (including all dimensions) of $Γ_{1}$ by the number of simplices of $Γ_{2}$ (including again all dimensions), which gives the following upper bound of $N$ :

N \leq n m 2^{2 d}

Now, using [3, Lemma 2.3.10] we can refine the polyhedral subdivision $Θ$ , without adding any new vertex, to get a triangulation of $K$ . By this way, we get a triangulation of $K$ with at most $n m 2^{2 d}$ vertices. Now, the Upper Bound Theorem (see [3, Corollary 2-6-5]) applies to this triangulation of $K$ , since $K$ is a topological ball of dimension $d$ . In particular, we get that

h (n, m) = f_{d} (C (n m 2^{2 d} + 1, d + 1)) - (d + 1)

satisfies the proposition. Here $f_{d} (H)$ denotes the number of faces of dimension $d$ (i.e., $d$ -simplices) of the simplicial complex $H$ and $C (N, d + 1)$ denotes the simplicial $d$ -sphere given by the proper faces of the cyclic $d + 1$ -polytope with $N$ vertices, which is the convex hull of $N$

arbitrary points taken from the moment curve

{(t, t^{2}, t^{3}, . . ., t^{d + 1}) : t \in R} \subseteq R^{d + 1}

$□$

Theorem 10.

Let $K \subseteq R^{d}$ be a convex polyhedron with non-empty interior, and let $0 = r_{0} < r_{1} < r_{2} < \dots < r_{k} < \dots$ , $0 = n_{0} < n_{1} \leq n_{2} \leq \dots \leq n_{k} \leq \dots$ be two sequences of natural numbers. Then the following holds true:

$(C (K), {S_{n, n}^{d} (K)})$ is an approximation scheme.
$(C (K), {S_{n, n}^{d} (K)})$ satisfies Shapiro’s theorem.
For any decreasing sequence ${ε_{n}}$ which converges to zero, there exists $f \in C (K)$ such that

$E (f, S_{r_{k}, n_{k}}^{d} (K))_{C (K)} \geq ε_{k} for all % k \in N$

Proof. It follows from Proposition 9 that there exists a function $h : N \times N \to N$ such that

S_{r_{k}, n_{k}}^{d} (K) + S_{r_{s}, n_{s}}^{d} (K) \subseteq S_{h (r_{k}, r_{s}), max {n_{k}, n_{s}}}^{d} (K),

which proves $(a)$ with jump function $J (n) = h (n, n)$ , since the other properties needed to be an approximation scheme are well known for these sets. To prove $(b)$ and $(c)$ we use a similar argument to the one used in the proof of Theorem 8. Concretely, given the convex polyhedron $K$ , we can assume with no loss of generality that $ρ = diam (K) = {max}_{x, y \in K} ∥ x - y ∥$ and

K \cap R \times {0} \times \dots \times {0} = [a, a + ρ] \times {0} \times \dots \times {0}

for certain $a \in R$ , since rotations and translations of the space transform triangulations into triangulations and preserve the spaces polynomials.

For each $ϕ \in S_{r, n}^{d} (K)$ , there exists a triangulation $Γ$ of $K$ with $# Γ \leq r$ such that

ϕ (x) = \sum Δ \in Γ p_{Δ} (x) χ_{Δ} (x) for all x \in K,

where $p_{Δ}$ denotes, for each $Δ \in Γ$ , a polynomial of total degree $deg (p) \leq n$ . Now, given the triangulation $Γ$ , it is clear that $Γ$ induces a triangulation (i.e., a partition) of $[a, a + ρ]$ which contains at most $2 r$ intervals, with the property that $ϕ (t, 0, \dots, 0) \in S_{2 r, n}^{1} ([a, a + ρ])$ . Hence, for any $f \in C (K)$ we have that

E (f, S_{r_{k}, n_{k}}^{d} (K))_{C (K)} \geq E (g, S_{2 r_{k}, n_{k}}^{1} ([a, a + ρ]))_{C [a, a + ρ]},

where $g (t) = f (t, 0, 0, \dots, 0)$ . Obviously, the convexity of $K$ and the fact that $ρ = diam (K)$ again imply that, given $g \in C [a, b]$ , the function $¯ ¯ ¯ g (x_{1}, \dots, x_{d}) = g (x_{1})$ belongs to $C (K)$ and $¯ ¯ ¯ g (t, 0, \dots, 0) = g (t)$ for all $t \in [a, a + ρ]$ . Hence, parts $(b)$ and $(c)$ follow from Theorem 7 $□$

Let us now demonstrate a negative result for approximation by feedforward neural networks with many layers. Let $σ \in C (R)$ be a nonpolynomial function and let us consider the approximation of continuous learning functions defined on compact subsets of $R^{d}$ with activation function $σ$ . We denote by $τ_{r, n}^{σ, d}$ the set of functions defined on $R^{d}$ computed by a feedforward neural network with at most $r$ layers and $n$ units in each layer, with activation function $σ$ . For example, an element of $τ_{1, n}^{σ, d}$ is a function of the form

ϕ (x) = n \sum i = 1 c_{i} σ (w^{i} \cdot x + b_{i}) where w^{i}, x \in R^{d} and c_{i}, b_{i} \in R

and an element of $τ_{2, n}^{σ, d}$ is a function of the form

ϕ (x) = n \sum i = 1 d_{i} σ (n \sum j = 1 c_{i j} σ (w^{i j} \cdot x + b_{i j}) + δ_{i}), where w^{i j}, x \in R^{d} and c_{i j}, b_{i j}, d_{i}, δ_{i} \in R

The following result holds:

Theorem 11.

Let $0 = r_{0} < r_{1} < r_{2} < \dots < r_{k} < \dots$ , $0 = n_{0} < n_{1} \leq n_{2} \leq \dots \leq n_{k} \leq \dots$ be two sequences of natural numbers and let ${ε_{n}}$ be any non-increasing sequence which converges to $0$ . Then:

If $K \subseteq R^{d}$ is compact and convex and $σ$ is a rational function, there exists $f \in C (K)$ such that

$E (f, τ_{r_{k}, n_{k}}^{σ, d})_{C (K)} \geq ε_{k} for all k = 1, 2, \dots .$
If $K \subseteq R^{d}$ is a convex polyhedron and $σ$ is a linear spline with finitely many pieces, there exists $f \in C (K)$ such that

$E (f, τ_{r_{k}, n_{k}}^{σ, d})_{C (K)} \geq ε_{k} for all k = 1, 2, \dots .$

In particular, this result applies to neural networks with activation functions $σ = ReLU$ and $σ = Hard Tanh$ .

Proof. If $σ$ is a (non polynomial) rational function of order $N$ then all elements of $τ_{r, n}^{σ, d}$ are also rational functions and their orders are uniformly bounded by a function $o (r, n, N)$ . Moreover, for the estimation of the errors $∥ f - R ∥_{C (K)}$ with $R \in τ_{r, n}^{σ, d}$ , we need to impose $R \in C (K)$ since, if $R$ has a pole on $K$ , then $∥ f - R ∥_{C (K)} = \infty$ and this function does not contribute to the estimation of the error $E (f, τ_{r, n}^{σ, d})_{C (K)}$ . Thus, we can claim that

E (f, τ_{r, n}^{σ, d})_{C (K)} \geq E (f, R_{o (r, n, N)}^{d} (K))_{C (K)}

and, taking $m_{k} = {max}_{i \leq k} o (r_{i}, n_{i}, N)$ , we have that

E (f, τ_{r_{k}, n_{k}}^{σ, d})_{C (K)} \geq E (f, R_{m_{k}}^{d} (K))_{C (K)}, k = 1, 2, \dots

and $(a)$ follows from Theorem 8.

Let us now prove $(b)$ . If $σ$ is a continuous linear spline with finitely many pieces, say $N$ , then every element of $τ_{r, n}^{σ, d}$ is again a continuous linear spline (in $d$ variables), defined on a finite set of polyhedra. For $r = 1$ we have that every element $ϕ \in τ_{1, n}^{σ, d}$ is of the form

ϕ (x) = n \sum i = 1 c_{i} σ (w^{i} \cdot x + b_{i}) with w^{i}, x \in R^{d} and c_{i}, b_{i} \in R,

where $σ (t) = A_{k} t + B_{k}$ for all $t \in I_{k}$ and ${I_{k}}_{k = 1}^{N}$ is a fixed partition of $R$ in $N$ intervals (two of them are semi-infinite and the others are finite intervals). This means that the evaluation of $ϕ$ takes the form:

ϕ (x) = n \sum i = 1 c_{i} (A_{k (i)} (w^{i} \cdot x + b_{i}) + B_{k (i)}) where% w^{i} \cdot x + b_{i} \in I_{k (i)},, i = 1, \dots, n,

which proves that $ϕ$ is piecewise linear and the number of pieces (where the function changes from one linear polynomial to another) is finite and controlled by a finite set of linear inequalities, so that each piece is a convex polyhedron with a well controlled number of faces. An induction argument -which takes into account the definition of neural networks of several layers- shows that all elements of $τ_{r, n}^{σ, d}$ belong to $S_{τ (N, r, n, d), 1}^{d} (K)$ for a certain fixed function $τ$ . The result follows from part $(c)$ of Theorem 10. $□$

References

[1] J. M. Almira, T. Oikhberg Approximation schemes satisfying Shapiro’s Theorem, Journal of Approximation Theory 164 (2012) 534-571.
[2] J. M. Almira, T. Oikhberg Shapiro’s theorem for subspaces, Journal of Mathematical Analysis and Applications, 388 (2012) 282-302.
[3] J. A. De Loera, J. Rambau and F. Santos, Triangulations. Structures for Algorithms and Applications, Springer, 2010.
[4] G.G. Lorentz, M. v. Golitschek, Y. Makovoz, Constructive Approximation. Advanced Problems, Springer, 1996.
[5] Y. Gordon, V. Maiorov, M. Meyer, S. Reisner, On the best approximation by ridge functions in the uniform norm, Constructive Approximation 18 (2002) 61-85.
[6] N. J. Guliyev, V. E. Ismailov, On the approximation by single hidden layer feedforward neural networks with fixed weights, Neural Networks, 98 (2018) 296-304.
[7] N. J. Guliyev, V. E. Ismailov, Approximation capability of two hidden layer feedforward neural networks with fixed weights, Neurocomputing, 316 (2018) 262-269.
[8] K. Hornik, M. Stinchcombe, H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989) 359-366.
[9] M. Leshno, V. Y. Lin, A. Pinkus, S. Schocken, Multilayer Feedforward Networks with a non polynomial activation function can approximate any function, Neural Networks 6 (1993) 861-867.
[10] V. Maiorov, A. Pinkus, Lower bounds for approximations by MLP neural networks, Neurocomputing 25 (1999) 81-91.
[11] V. Kůrková, Kolmogorov’s theorem and multilayer neural networks, Neural Networks 5 (1992) 501-506.
[12] V. Kůrková, Kolmogorov’s theorem is relevant, Neural Computation 3 (1991) 617-622.
[13] A. Pinkus, Approximation theory of the MLP model in Neural Networks, Acta Numerica 8 (1999), 143-196.
[14] K-Y. Siu, V. P. Roychowdhury, and T. Kailath, Rational Approximation Techniques for Analysis of Neural Networks IEEE Transactions of Inf. Theory 40 (2) (1994) 455-466.
[15] M. Telgarsky,
Neural networks and rational functions, Proc. Machine Learning Research ICML (2017) 3387-3393.
[16] M. Telgarsky, Benefits of depth in neural networks, JMLR: Workshop and Conference Proceedings vol 49 (2016) 1-23.
[17] R. C. Williamson, Rational parametrization of neural networks, Advances in Neural Inf. Processing Systems 6 (1993) 623-630.
[18] R. C. Williamson, P. L. Barlett, Splines, rational functions and neural networks, Advances in Neural Inf. Processing Systems 5 (1992) 1040-1047.

Some negative results for Neural Networks

Authors

Approximation of functions by neural networks

Optimal Approximation with Sparse Neural Networks and Applications

Neural Network Processing Neural Networks: An efficient way to learn higher order functions

Negative type diversities, a multi-dimensional analogue of negative type metrics

Negative Learning Rates and P-Learning

Application of neural networks to classification of data of the TUS orbital telescope

On the Computational Efficiency of Training Neural Networks

1. Introduction

2. A general negative result for feedforward neural networks with one hidden layer

Proposition 1.

Theorem 2 (Almira and Oikhberg, 2012).

Remark 3.

Theorem 4.

Theorem 5 (Gordon, Maiorov, Meyer and Reisner, 2002).

3. Negative results for feedforward neural networks with several hidden layers

Theorem 6.

Theorem 7.

Theorem 8.

Proposition 9.

Theorem 10.

Theorem 11.

References

Some negative results for Neural Networks

Authors

Related Research

Approximation of functions by neural networks

Optimal Approximation with Sparse Neural Networks and Applications

Neural Network Processing Neural Networks: An efficient way to learn higher order functions

Negative type diversities, a multi-dimensional analogue of negative type metrics

Negative Learning Rates and P-Learning

Application of neural networks to classification of data of the TUS orbital telescope

On the Computational Efficiency of Training Neural Networks

This week in AI

1. Introduction

2. A general negative result for feedforward neural networks with one hidden layer

Proposition 1.

Theorem 2 (Almira and Oikhberg, 2012).

Remark 3.

Theorem 4.

Theorem 5 (Gordon, Maiorov, Meyer and Reisner, 2002).

3. Negative results for feedforward neural networks with several hidden layers

Theorem 6.

Theorem 7.

Theorem 8.

Proposition 9.

Theorem 10.

Theorem 11.

References