Deep learning, in the form of artificial neural networks, has seen a dramatic resurgence in the past recent years, achieving great performance improvements in various fields of artificial intelligence such as computer vision and speech recognition. While empirically successful, our theoretical understanding of deep learning is still limited at best.
An emerging line of recent works has studied the expressive power of neural networks: What functions can and cannot be represented by networks of a given architecture (see related work section below). A particular focus has been the trade-off between the network’s width and depth: On the one hand, it is well-known that large enough networks of depth can already approximate any continuous target function on to arbitrary accuracy (Cybenko, 1989; Hornik, 1991). On the other hand, it has long been evident that deeper networks tend to perform better than shallow ones, a phenomenon supported by the intuition that depth, providing compositional expressibility, is necessary for efficiently representing some functions. Indeed, recent empirical evidence suggests that even at large depths, deeper networks can offer benefits over shallower networks (He et al., 2015).
To demonstrate the power of depth in neural networks, a clean and precise approach is to prove the existence of functions which can be expressed (or well-approximated) by moderately-sized networks of a given depth, yet cannot be approximated well by shallower networks, even if their size is much larger. However, the mere existence of such functions is not enough: Ideally, we would like to show such depth separation results using natural, interpretable functions, of the type we may expect neural networks to successfully train on. Proving that depth is necessary for such functions can give us a clearer and more useful insight into what various neural network architectures can and cannot express in practice.
In this paper, we provide several contributions to this emerging line of work. We focus on standard, vanilla feedforward networks (using some fixed activation function, such as the popular ReLU), and measure expressiveness directly in terms of approximation error, defined as the expected squared loss with respect to some distribution over the input domain. In this setting, we show the following:
We prove that the indicator of the Euclidean unit ball, in , which can be easily approximated to accuracy using a 3-layer network with neurons, cannot be approximated to an accuracy higher than using a 2-layer network, unless its width is exponential in . In fact, we show the same result more generally, for any indicator of an ellipsoid (where is a non-singular matrix and
is a vector). The proof is based on a reduction from the main result ofEldan & Shamir (2016), which shows a separation between 2-layer and 3-layer networks using a more complicated and less natural radial function.
We show that this depth/width trade-off can also be observed experimentally: Specifically, that the indicator of a unit ball can be learned quite well using a 3-layer network, using standard backpropagation, but learning the same function with a 2-layer network (even if much larger) is significantly more difficult. Our theoretical result indicates that this gap in performance is due to approximation error issues. This experiment also highlights the fact that our separation result is for a natural function that is not just well-approximated by some 3-layer network, but can also be learned well from data using standard methods.
We prove that any radial function , where and is piecewise-linear, cannot be approximated to accuracy by a depth 2 ReLU network of width less than . In contrast, such functions can be represented exactly by 3-layer ReLU networks.
Finally, we prove that any member of a wide family of non-linear and twice-differentiable functions (including for instance in ), which can be approximated to accuracy using ReLU networks of depth and width , cannot be approximated to similar accuracy by constant-depth ReLU networks, unless their width is at least . We note that a similar result appeared online concurrently and independently of ours in Yarotsky (2016); Liang & Srikant (2016), but the setting is a bit different (see related work below for more details).
The question of studying the effect of depth in neural network has received considerable attention recently, and studied under various settings. Many of these works consider a somewhat different setting than ours, and hence are not directly comparable. These include networks which are not plain-vanilla ones (e.g. Cohen et al. (2016); Delalleau & Bengio (2011); Martens & Medabalimi (2014)), measuring quantities other than approximation error (e.g. Bianchini & Scarselli (2014); Poole et al. (2016)), focusing only on approximation upper bounds (e.g. Shaham et al. (2016)), or measuring approximation error in terms of -type bounds, i.e. rather than -type bounds (e.g. Yarotsky (2016); Liang & Srikant (2016)). We note that the latter distinction is important: Although bounds are more common in the approximation theory literature,
bounds are more natural in the context of statistical machine learning problems (where we care about the expected loss over some distribution). Moreover,approximation lower bounds are stronger, in the sense that an lower bound easily translates to a lower bound on lower bound, but not vice versa111To give a trivial example, ReLU networks always express continuous functions, and therefore can never approximate a discontinuous function such as in an sense, yet can easily approximate it in an sense given any continuous distribution..
A noteworthy paper in the same setting as ours is Telgarsky (2016), which proves a separation result between the expressivity of ReLU networks of depth and depth (for any ). This holds even for one-dimensional functions, where a depth network is shown to realize a saw-tooth function with oscillations, whereas any network of depth would require a width super-polynomial in to approximate it by more than a constant. In fact, we ourselves rely on this construction in the proofs of our results in Sec. 5. On the flip side, in our paper we focus on separation in terms of the accuracy or dimension, rather than a parameter . Moreover, the construction there relies on a highly oscillatory function, with Lipschitz constant exponential in almost everywhere. In contrast, in our paper we focus on simpler functions, of the type that are likely to be learnable from data using standard methods.
Our separation results in Sec. 5 (for smooth non-linear functions) are closely related to those of Yarotsky (2016); Liang & Srikant (2016), which appeared online concurrently and independently of our work, and the proof ideas are quite similar. However, these papers focused on bounds rather than bounds. Moreover, Yarotsky (2016) considers a class of functions different than ours in their positive results, and Liang & Srikant (2016) consider networks employing a mix of ReLU and threshold activations, whereas we consider a purely ReLU network.
Another relevant and insightful work is Poggio et al. (2016), which considers width vs. depth and provide general results on expressibility of functions with a compositional nature. However, the focus there is on worse-case approximation over general classes of functions, rather than separation results in terms of specific functions as we do here, and the details and setting is somewhat orthogonal to ours.
In general, we let bold-faced letters such as denote vectors, and capital letters denote matrices or probabilistic events. denotes the Euclidean norm, and the -norm. denotes the indicator function. We use the standard asymptotic notation and to hide constants, and and to hide constants and factors logarithmic in the problem parameters.
Neural Networks. We consider feed-forward neural networks, computing functions from to . The network is composed of layers of neurons, where each neuron computes a function of the form , where is a weight vector, is a bias term and is a non-linear activation function, such as the ReLU function . Letting be a shorthand for , we define a layer of neurons as . By denoting the output of the layer as , we can define a network of arbitrary depth recursively by , where represent the matrix of weights and bias of the layer, respectively. Following a standard convention for multi-layer networks, the final layer is a purely linear function with no bias, i.e. . We define the depth of the network as the number of layers , and denote the number of neurons in the layer as the size of the layer. We define the width of a network as . Finally, a ReLU network is a neural network where all the non-linear activations are the ReLU function. We use “2-layer” and “3-layer” to denote networks of depth 2 and 3. In particular, in our notation a 2-layer ReLU network has the form
for some parameters and -dimensional vectors . Similarly, a 3-layer ReLU network has the form
for some parameters .
Approximation error. Given some function on a domain
endowed with some probability distribution (with density function), we define the quality of its approximation by some other function as . We refer to this as approximation in the -norm sense. In one of our results (Thm. 6), we also consider approximation in the -norm sense, defined as . Clearly, this upper-bounds the (square root of the) approximation error defined above, so as discussed in the introduction, lower bounds on the approximation error (w.r.t. any distribution) are stronger than lower bounds on the approximation error.
3 Indicators of Balls and Ellipsoids
We begin by considering one of the simplest possible function classes on , namely indicators of balls (and more generally, ellipsoids). The ability to compute such functions is necessary for many useful primitives, for example determining if the distance between two points in Euclidean space is below or above some threshold (either with respect to the Euclidean distance, or a more general Mahalanobis distance). In this section, we show a depth separation result for such functions: Although they can be easily approximated with 3-layer networks, no 2-layer network can approximate it to high accuracy w.r.t. any distribution, unless its width is exponential in the dimension. This is formally stated in the following theorem:
Theorem 1 (Inapproximability with 2-layer networks).
The following holds for some positive universal constants , and any network employing an activation function satisfying Assumptions 1 and 2 in Eldan & Shamir (2016): For any , and any non-singular matrix , and , there exists a continuous probability distribution on , such that for any function computed by a 2-layer network of width at most , and for the function , we have
We note that the assumptions from Eldan & Shamir (2016) are very mild, and apply to all standard activation functions, including ReLU, sigmoid and threshold.
The formal proof of Thm. 1 (provided below) is based on a reduction from the main result of Eldan & Shamir (2016), which shows the existence of a certain radial function (depending on the input only through its norm) and a probability distribution which cannot be expressed by a 2-layer network, whose width is less than exponential in the dimension to more than constant accuracy. A closer look at the proof reveals that this function (denoted as ) can be expressed as a sum of indicators of balls of various radii. We argue that if we could have accurately approximated a given ball indicator with respect to all distributions, then we could have approximated all the indicators whose sum add up to
, and hence reach a contradiction. By a linear transformation argument, we show the same contradiction would have occured if we could have approximated the indicators of an non-degenerate ellipse with respect to any distribution. The formal proof is provided below:
Proof of Thm. 1.
Assume by contradiction that for as described in the theorem, and for any distribution , there exists a two-layer network of width at most , such that
Let and be a non-singular matrix and vector respectively, to be determined later. We begin by performing a change of variables, , , which yields
In particular, let us choose the distribution defined as , where is the (continuous) distribution used in the main result of Eldan & Shamir (2016) (note that is indeed a distribution, since , which by the change of variables , equals ). Plugging the definition of in Eq. (1), and using the fact that , we get
Letting be an arbitrary parameter, we now pick and . Recalling the definition of as , we get that
Note that expresses a 2-layer network composed with a linear transformation of the input, and hence can be expressed in turn by a 2-layer network (as we can absorb the linear transformation into the parameters of each neuron in the first layer). Therefore, letting denote the norm in function space, we showed the following: For any , there exists a 2-layer network such that
With this key result in hand, we now turn to complete the proof. We consider the function from Eldan & Shamir (2016), for which it was proven that no 2-layer network can approximate it w.r.t. to better than constant accuracy, unless its width is exponential in the dimension . In particular can be written as
where are disjoint intervals, , and where is the dimension. Since can also be written as
we get by Eq. (4) and the triangle inequality that
However, since a linear combination of 2-layer neural networks of width at most is still a 2-layer network, of width at most , we get that is a 2-layer network, of width at most , which approximates to an accuracy of less than . Hence, by picking sufficiently small, we get a contradiction to the result of Eldan & Shamir (2016), that no 2-layer network of width smaller than (for some constant ) can approximate to more than constant accuracy, for a sufficiently large dimension . ∎
To complement Thm. 1, we also show that such indicator functions can be easily approximated with 3-layer networks. The argument is quite simple: Using an activation such as ReLU or Sigmoid, we can use one layer to approximate any Lipschitz continuous function on any bounded interval, and in particular . Given a vector , we can apply this construction on each coordinate seperately, hence approximating . Similarly, we can approximate for arbitrary fixed matrices and vectors . Finally, with a 3-layer network, we can use the second layer to compute a continuous approximation to the threshold function . Composing these two layers, we get an arbitrarily good approximation to the function w.r.t. any continuous distribution, with the network size scaling polynomially with the dimension and the required accuracy. In the theorem below, we formalize this intuition, where for simplicity we focus on approximating the indicator of the unit ball:
Theorem 2 (Approximability with 3-layer networks).
Given , for any activation function satisfying Assumption 1 in Eldan & Shamir (2016) and any continuous probability distribution on , there exists a constant dependent only on , and a function expressible by a 3-layer network of width at most , such that the following holds:
where is a constant depending solely on .
The proof of the theorem appears in Subsection 6.1.
3.1 An Experiment
In this subsection, we empirically demonstrate that indicator functions of balls are indeed easier to learn with a 3-layer network, compared to a 2-layer network (even if the 2-layer network is significantly larger). This indicates that the depth/width trade-off for indicators of balls, predicted by our theory, can indeed be observed experimentally. Moreover, it highlights the fact that our separation result is for simple natural functions, that can be learned reasonably well from data using standard methods.
For our experiment, we sampled data instances in , with a direction chosen uniformly at random and a norm drawn uniformly at random from the interval . To each instance, we associated a target value computed according to the target function . Another examples were generated in a similar manner and used as a validation set.
We trained ReLU networks on this dataset:
One 3-layer network, with a first hidden layer of size , a second hidden layer of size , and a linear output neuron.
Four 2-layer networks, with hidden layer of sizes and , and a linear output neuron.
Training was performed with backpropagation, using the TensorFlow library. We used the squared lossand batches of size 100. For all networks, we chose a momentum parameter of 0.95, and a learning rate starting at 0.1, decaying by a multiplicative factor of 0.95 every 1000 batches, and stopping at .
The results are presented in Fig. 1. As can be clearly seen, the 3-layer network achieves significantly better performance than the 2-layer networks. This is true even though some of these networks are significantly larger and with more parameters (for example, the 2-layer, width 800 network has ~80K parameters, vs. ~10K parameters for the 3-layer network). This gap in performance is the exact opposite of what might be expected based on parameter counting alone. Moreover, increasing the width of the 2-layer networks exhibits diminishing returns: The performance improvement in doubling the width from 100 to 200 is much larger than doubling the width from 200 to 400 or 400 to 800. This indicates that one would need a much larger 2-layer network to match the 3-layer, width 100 network’s performance. Thus, we conclude that the network’s depth indeed plays a crucial role, and that 3-layer networks are inherently more suitable to express indicator functions of the type we studied.
4 Radial Functions; ReLU Networks
Having considered functions depending on the norm, we now turn to consider functions depending on the norm. Focusing on ReLU networks, we will show a certain separation result holding for any non-linear function, which depends on the input only via its 1-norm .
Let be a function such that for some and ,
Then there exists a distribution over , such that if a 2-layer ReLU network satisfies
then its width must be at least (where the notation hides constants and factors logarithmic in ).
To give a concrete example, suppose that , which cannot be approximated by a linear function better than in an -neighborhood of . By taking and , we get that no 2-layer network can approximate the function (at least with respect to some distribution), unless its width is . On the flip side, can be expressed exactly by a 3-layer, width ReLU network: , where the output neuron is simply the identity function. The same argument would work for any piecewise-linear . More generally, the same kind of argument would work for any function exhibiting a non-linear behavior at some points: Such functions can be well-approximated by 3-layer networks (by approximating with a piecewise-linear function), yet any approximating 2-layer network will have a lower bound on its size as specified in the theorem.
Intuitively, the proof relies on showing that any good -layer approximation of must capture the non-linear behavior of close to “most” points satisfying . However, a -layer ReLU network is piecewise linear, with non-linearities only at the union of the hyperplanes . This implies that “most” points s.t. must be -close to a hyperplane . However, the geometry of the ball is such that the neighborhood of any single hyperplane can only cover a “small” portion of that ball, yet we need to cover most of the ball. Using this and an appropriate construction, we show that required number of hyperplanes is at least , as long as (and if is smaller than that, we can simply use one neuron/hyperplane for each of the facets of the ball, and get a covering using neurons/hyperplanes). The formal proof appears in Subsection 6.2.
We note that the bound in Thm. 3 is of a weaker nature than the bound in the previous section, in that the lower bound is only polynomial rather than exponential (albeit w.r.t. different problem parameters: vs. ). Nevertheless, we believe this does point out that
balls also pose a geometric difficulty for 2-layer networks, and conjecture that our lower bound can be considerably improved: Indeed, at the moment we do not know how to approximate a function such aswith 2-layer networks to better than constant accuracy, using less than neurons.
5 Nonlinear Functions; ReLU Networks
In this section, we establish a depth separation result for approximating continuously twice-differentiable (
) functions using ReLU neural networks. Unlike the previous results in this paper, the separation is for depths which can be larger than 3, depending on the required approximation error. Also, the results will all be with respect to the uniform distributionover . As mentioned earlier, the results and techniques in this section are closely related to the independent results of Yarotsky (2016); Liang & Srikant (2016), but our emphasis is on rather than approximation bounds, and we focus on somewhat different network architectures and function classes.
Clearly, not all functions are difficult to approximate (e.g. a linear function can be expressed exactly with a 2-layer network). Instead, we consider functions which have a certain degree of non-linearity, in the sense that its Hessians are non-zero along some direction, on a significant portion of the domain. Formally, we make the following definition:
Let denote the uniform distribution on . For a function and some , denote
where is the -dimensional unit hypersphere, and is the set of all connected and measurable subsets of .
In words, is the measure (w.r.t. the uniform distribution on ) of the largest connected set in the domain of , where at any point, has curvature at least along some fixed direction . The “prototypical” functions we are interested in is when is lower bounded by a constant (e.g. it is if is strongly convex). We stress that our results in this section will hold equally well by considering the condition as well, however for the sake of simplicity we focus on the former condition appearing in Def. 1. Our goal is to show a depth separation result inidividually for any such function (that is, for any such function, there is a gap in the attainable error between deeper and shallower networks, even if the shallow network is considerably larger).
As usual, we start with an inapproximability result. Specifically, we prove the following lower bound on the attainable approximation error of , using a ReLU neural network of a given depth and width:
For any function , any , and any function on expressible by a ReLU network of depth and maximal width , it holds that
where is a universal constant.
The theorem conveys a key tradeoff between depth and width when approximating a function using ReLU networks: The error cannot decay faster than polynomially in the width , yet the bound deteriorates exponentially in the depth . As we show later on, this deterioration does not stem from the looseness in the bound: For well-behaved , it is indeed possible to construct ReLU networks, where the approximation error decays exponentially with depth.
The proof of Thm. 4 appears in Subsection 6.3, and is based on a series of intermediate results. First, we show that any strictly curved function (in a sense similar to Definition 1) cannot be well-approximated in an sense by piecewise linear functions, unless the number of linear regions is large. To that end, we first establish some necessary tools based on Legendre polynomials. We then prove a result specific to the one-dimensional case, including an explicit lower bound if the target function is quadratic (Thm. 9) or strongly convex or concave (Thm. 10). We then expand the construction to get an error lower bound in general dimension , depending on the number of linear regions in the approximating piecewise-linear function. Finally, we note that any ReLU network induces a piecewise-linear function, and bound the number of linear regions induced by a ReLU network of a given width and depth (using a lemma borrowed from Telgarsky (2016)). Combining this with the previous lower bound yields Thm. 4.
We now turn to complement this lower bound with an approximability result, showing that with more depth, a wide family of functions to which Thm. 4 applies can be approximated with exponentially high accuracy. Specifically, we consider functions which can be approximated using a moderate number of multiplications and additions, where the values of intermediate computations are bounded (for example, a special case is any function approximable by a moderately-sized Boolean circuit, or a polynomial).
The key result to show this is the following, which implies that the multiplication of two (bounded-size) numbers can be approximated by a ReLU network, with error decaying exponentially with depth:
Let , and let be arbitrary. Then exists a ReLU neural network of width and depth satisfying
The idea of the construction is that depth allows us to compute highly-oscillating functions, which can extract high-order bits from the binary representation of the inputs. Given these bits, one can compute the product by a procedure resembling long multiplication, as shown in Fig. 2, and formally proven as follows:
Proof of Thm. 5.
We begin by observing that by using a simple linear change of variables on , we may assume without loss of generality that , as we can just rescale to the interval , and then map it back to its original domain , where the error will multiply by a factor of . Then by requiring accuracy instead of , the result will follow.
The key behind the proof is that performing bit-wise operations on the first bits of
yields an estimation of the product to accuracy. Let be the binary representation of where is the bit of , then
Eq. (5) implies
Requiring that , it suffices to show the existence of a network which approximates the function to accuracy , where . This way both approximations will be at most , resulting in the desired accuracy of .
Before specifying the architecture which extracts the bit of , we first describe the last 2 layers of the network. Let the penultimate layer comprise of neurons, each receiving both and as input, and having the set of weights . Thus, the output of the neuron in the penultimate layer is
Let the final single output neuron have the set of weights , this way, the output of the network will be as required.
We now specify the architecture which extracts the first most significant bits of . In Telgarsky (2016), the author demonstrates how the composition of the function
with itself times, , yields a highly oscillatory triangle wave function in the domain . Furthermore, we observe that , and thus . Now, a linear shift of the input of by , and composing the output with
which converges to as , results in an approximation of :
We stress that choosing such that the network approximates the bit-wise product to accuracy will require to be of magnitude , but this poses no problem as representing such a number requires bits, which is also the magnitude of the size of the network, as suggested by the following analysis.
Next, we compute the size of the network required to implement the above approximation. To compute only two neurons are required, therefore can be computed using layers with neurons in each, and finally composing this with requires a subsequent layer with more neurons. To implement the bit extractor we therefore require a network of size . Using dummy neurons to propagate the bit for , the architecture extracting the most significant bits of will be of size . Adding the final component performing the multiplication estimation will require more layers of width and respectively, and an increase of the width by to propagate to the penultimate layer, resulting in a network of size . ∎
Thm. 5 shows that multiplication can be performed very accurately by deep networks. Moreover, additions can be computed by ReLU networks exactly, using only a single layer with neurons: Let be arbitrary, then is given in terms of ReLU summation by
Repeating these arguments, we see that any function which can be approximated by a bounded number of operations involving additions and multiplications, can also be approximated well by moderately-sized networks. This is formalized in the following theorem, which provides an approximation error upper bound (in the sense, which is stronger than for upper bounds):
Let be the family of functions on the domain with the property that is approximable to accuracy with respect to the infinity norm, using at most operations involving weighted addition, , where are fixed; and multiplication, , where each intermediate computation stage is bounded in the interval . Then there exists a universal constant , and a ReLU network of width and depth at most , such that
Suppose , where and . Then approximating to accuracy in the norm using a fixed depth ReLU network requires width at least , whereas there exists a ReLU network of depth and width at most which approximates to accuracy in the infinity norm, where is a polynomial depending solely on .
This research is supported in part by an FP7 Marie Curie CIG grant, Israel Science Foundation grant 425/13, and the Intel ICRI-CI Institute. We would like to thank Shai Shalev-Shwartz for some illuminating discussions, and Eran Amar for his valuable help with the experiment.
Bianchini & Scarselli (2014)
Bianchini, M. and Scarselli, F.
On the complexity of shallow and deep neural network classifiers.In ESANN, 2014.
Cohen et al. (2016)
Cohen, Nadav, Sharir, Or, and Shashua, Amnon.
On the expressive power of deep learning: A tensor analysis.In 29th Annual Conference on Learning Theory, pp. 698–728, 2016.
Approximation by superpositions of a sigmoidal function.Mathematics of control, signals and systems, 2(4):303–314, 1989.
- Delalleau & Bengio (2011) Delalleau, O. and Bengio, Y. Shallow vs. deep sum-product networks. In NIPS, pp. 666–674, 2011.
- Eldan & Shamir (2016) Eldan, Ronen and Shamir, Ohad. The power of depth for feedforward neural networks. In 29th Annual Conference on Learning Theory, pp. 907–940, 2016.
- He et al. (2015) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
- Hornik (1991) Hornik, Kurt. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251–257, 1991.
- Liang & Srikant (2016) Liang, Shiyu and Srikant, R. Why deep neural networks? arXiv preprint arXiv:1610.04161, 2016.
- Martens & Medabalimi (2014) Martens, J. and Medabalimi, V. On the expressive efficiency of sum product networks. arXiv preprint arXiv:1411.7717, 2014.
- Poggio et al. (2016) Poggio, Tomaso, Mhaskar, Hrushikesh, Rosasco, Lorenzo, Miranda, Brando, and Liao, Qianli. Why and when can deep–but not shallow–networks avoid the curse of dimensionality: a review. arXiv preprint arXiv:1611.00740, 2016.
- Poole et al. (2016) Poole, Ben, Lahiri, Subhaneil, Raghu, Maithreyi, Sohl-Dickstein, Jascha, and Ganguli, Surya. Exponential expressivity in deep neural networks through transient chaos. In Advances In Neural Information Processing Systems, pp. 3360–3368, 2016.
- Shaham et al. (2016) Shaham, Uri, Cloninger, Alexander, and Coifman, Ronald R. Provable approximation properties for deep neural networks. Applied and Computational Harmonic Analysis, 2016.
- Telgarsky (2016) Telgarsky, Matus. Benefits of depth in neural networks. arXiv preprint arXiv:1602.04485, 2016.
- Yarotsky (2016) Yarotsky, Dmitry. Error bounds for approximations with deep relu networks. arXiv preprint arXiv:1610.01145, 2016.
6.1 Proof of Thm. 2
This proof bears resemblance to the proof provided in Eldan & Shamir (2016)[Lemma 10], albeit once approximating , the following construction takes a slightly different route. For completeness, we also state assumption 1 from Eldan & Shamir (2016):
Given the activation function , there is a constant (depending only on ) such that the following holds: For any -Lipschitz function which is constant outside a bounded interval , and for any , there exist scalars , where , such that the function
As discussed in Eldan & Shamir (2016), this assumption is satisfied by ReLU, sigmoid, threshold, and more generally all standard activation functions we are familiar with.
Consider the -Lipschitz function
which is constant outside , as well as the function
on . Applying assumption 1, we obtain a function having the form so that
and where the width parameter is at most . Consequently, the function
can be expressed in the form where , yielding an approximation satisfying
We now invoke assumption 1 again to approximate the -Lipschitz function
and obtain an approximation satisfying
Now consider the composition , where is to be determined later. This composition has the form
for appropriate scalars and vectors , and where is at most . It is now left to bound the approximation error obtained by . Define for any ,
Since is continuous, there exists such that
Now, for any such that we have
Assuming , we have the above is at least
Taking , we get
For any satisfying . A similar argument shows that for any satisfying we have
Where the first summand in the penultimate inequality is justified due to being bounded in the interval by Eq. (6), and assuming