Over the past several years, neural nets particularly deep nets
have become the state of the art in a remarkable number of machine learning problems, from mastering Go to image recognition/segmentation and machine translation (see the review article for more background). Despite all their practical successes, a robust theory of why they work so well is in its infancy. Much of the work to date has focused on the problem of explaining and quantifying the expressivity the ability to approximate a rich class of functions of deep neural nets [1, 8, 9, 11, 12, 13, 15, 16, 17, 18]. Expressivity can be seen both as an effect of both depth and width. It has been known since at least the work of Cybenko  and Hornik-Stinchcombe-White  that if no constraint is placed on the width of a hidden layer, then a single hidden layer is enough to approximate essentially any function. The purpose of this article, in contrast, is to investigate the “effect of depth without the aid of width.” More precisely, for each we would like to estimate
In Theorem 1, we prove that This raises two questions:
Is the estimate in the previous line sharp?
How efficiently can nets of a given width approximate a given continuous function of variables?
On the subject of Q1, we will prove in forthcoming work with M. Sellke  that in fact When , the lower bound is simple to check, and the upper bound follows for example from Theorem 3.1 in . The main results in this article, however, concern Q1 and Q2 for convex functions. For instance, we prove in Theorem 1 that
This illustrates a central point of the present paper: the convexity of the activation makes nets well-adapted to representing convex functions on
Theorem 1 also addresses Q2 by providing quantitative estimates on the depth of a net with width that approximates a given convex function. We provide similar depth estimates for arbitrary continuous functions on but this time for nets of width Several of our depth estimates are based on the work of Balázs-György-Szepesvári  on max-affine estimators in convex regression.
In order to prove Theorem 1, we must understand what functions can be exactly computed by a net. Such functions are always piecewise affine, and we prove in Theorem 2 the converse: every piecewise affine function on can be exactly represented by a net with hidden layer width at most . Moreover, we prove that the depth of the network that computes such a function is bounded by the number affine pieces it contains. This extends the results of Arora-Basu-Mianjy-Mukherjee (e.g. Theorem 2.1 and Corollary 2.2 in ).
Convex functions again play a special role. We show that every convex function on that is piecewise affine with pieces can be represented exactly by a net with width and depth
2. Statement of Results
To state our results precisely, we set notation and recall several definitions. For and a continuous function write
Further, denote by
the modulus of continuity of whose value at is the maximum changes when its argument moves by at most Note that by definition of a continuous function, as Next, given and we define a feed-forward neural net with ReLU activations, input dimension , hidden layer width , depth and output dimension to be any member of the finite-dimensional family of functions
that map to In (4),
are affine transformations, and for every
We often denote such a net by and write
for the function it computes. Our first result contrasts both the width and depth required to approximate continuous, convex, and smooth functions by nets.
Let and be a positive function with . We have the following three cases:
- 1. ( is continuous):
There exists a sequence of feed-forward neural nets with ReLU activations, input dimension hidden layer width output dimension such that
In particular, Moreover, write for the modulus of continuity of and fix There exists a feed-forward neural nets with ReLU activations, input dimension hidden layer width output dimension and
- 2. ( is convex):
There exists a sequence of feed-forward neural nets with ReLU activations, input dimension hidden layer width and output dimension such that
Hence, Further, there exists such that if is both convex and Lipschitz with Lipschitz constant then the nets in (8) can be taken to satisfy
- 3. ( is smooth):
There exists a constant depending only on and a constant depending only on the maximum of the first derivative of such that for every the width nets in (5) can be chosen so that
The main novelty of Theorem 1 is the width estimate and the quantitative depth estimates (9) for convex functions as well as the analogous estimates (6) and (7) for continuous functions. Let us breifly explain the origin of the other estimates. The relation (5) and the corresponding estimate are a combination of the well-known fact that nets with one hidden layer can approximate any continuous function and a simple procedure by which a net with input dimension and a single hidden layer of width can be replaced by another net that computes the same function but has depth and width For these width nets, we are unaware of how to obtain quantitative estimates on the depth required to approximate a fixed continuous function to a given precision. At the expense of changing the width of our nets from to however, we furnish the estimates (6) and (7). On the other hand, using Theorem 3.1 in , when is sufficiently smooth, we obtain the depth estimates (10) for width nets.
Our next result concerns the exact representation of piecewise affine functions by nets. Instead of measuring the complexity of a such a function by its Lipschitz constant or modulus of continuity, the complexity of a piecewise affine function can be thought of as the minimal number of affine pieces needed to define it.
Let and be the function computed by some net with input dimension , output dimension and arbitrary width. There exist affine functions such that can be written as the difference of positive convex functions:
Moreover, there exists a feed-forward neural net with ReLU activations, input dimension hidden layer width output dimension and
that computes exactly. Finally, if is convex (and hence vanishes), then the width of can be taken to be and the depth can be taken to
The fact that the function computed by a net can be written as (11) follows from Theorem 2.1 in . The novelty in Theorem 2 is therefore the uniform width estimate in the representation on any function computed by a net and the width estimate for convex functions. Theorem 2 will be used in the proof of Theorem 1.
3. Relation to Previous Work
This article is related to several strands of prior work:
Theorems 1-2 are “deep and narrow” analogs of the well-known “shallow and wide” universal approximation results (e.g. Cybenko  and Hornik- Stinchcombe -White ) for feed-forward neural nets. Those articles show that essentially any scalar function on the dimensional unit cube can be arbitrarily well-approximated by a feed-forward neural net with a single hidden layer with arbitrary width. Such results hold for a wide class of nonlinear activations but are not particularly illuminating from the point of understanding the expressive advantages of depth in neural nets.
The results in this article complement the work of Liao-Mhaskar-Poggio  and Mhaskar-Poggio , who consider the advantages of depth for representing certain heirarchical or compositional functions by neural nets with both and non- activations. Their results (e.g. Theorem 1 in  and Theorem 3.1 in ) give bounds on the width for approximation both for shallow and certain deep heirarchical nets.
Theorems 1-2 are also quantitative analogs of Corollary 2.2 and Theorem 2.4 in the work of Arora-Basu-Mianjy-Mukerjee . Their results give bounds on the depth of a net needed to compute exactly a piecewise linear function of variables. However, except when
they do not obtain an estimate on the number of neurons in such a network and hence cannot bound the width of the hidden layers.
Our results are related to Theorems II.1 and II.4 of Rolnick-Tegmark , which are themselves extensions of Lin-Rolnick-Tegmark . Their results give lower bounds on the total size (number of neurons) of a neural net (with non- activations) that approximates sparse multivariable polynomials. Their bounds do not imply a control on the width of such networks that depends only on the number of variables, however.
This work was inpsired in part by questions raised in the work of Telgarsky [15, 16, 17]. In particular, in Theorems 1.1 and 1.2 of , Telgarsky constructs interesting examples of sawthooth functions that can be computed efficiently by deep width nets that cannot be well-approximated by shallower networks with a simlar number of parameters.
Theorems 1-2 are quantitative statements about the expressive power of depth without the aid of width. This topic, usually without considering bounds on the width, has been taken up by many authors. We refer the reader to [12, 13] for several interesting quantitative measures of the complexity of functions computed by deep neural nets.
Finally, we refer the reader to the interesting work of Yarofsky , which provides bounds on the total number of parameters in a net needed to approximate a given class of functions (mainly balls in various Sobolev spaces).
It is a pleasure to thank Elchanan Mossel and Leonid Hanin for many helpful discussions. This paper originated while I attended EM’s class on deep learning. In particular, I would like to thank him for suggesting proving quantitative bounds in Theorem 2 and for suggesting that a lower bound can be obtained by taking piece-wise linear functions with many different directions. He also pointed out that the width estimates for continuous function in Theorem 1 where sub-optimal in a previous draft. l would also like to thank Leonid Hanin for detailed comments on a several previous drafts and for useful references to results in approximation theory. I am also grateful to Brandon Rule and Matus Telgarsky for comments on an earlier version of this article. I am also grateful to BR for the original suggestion to investigate the expressivity of neural nets of width . I also would like to thank Max Kleiman-Weiner for useful comments and discussion. Finally, I thank Zhou Lu for pointing out a serious error what used to be Theorem 3 in a previous version of this article. I have removed that result.
5. Proof of Theorem 2
We first treat the case
when is convex. We seek to show that can be exactly represented by a net with input dimension hidden layer width , and depth Our proof relies on the following observation.
Fix let be an arbitrary function, and be affine. Define an invertible affine transformation by
Then the image of the graph of under
is the graph of viewed as a function on
We have Hence, for each we have
We now construct a neural net that computes Define invertible affine functions by
where is the
st standard basis vector so thatis the linear map from to that maps to Finally, set
where maps to the graph of the zero function. Note that the in this initial layer is linear. With this notation, repeatedly using Lemma 3, we find that
therefore has input dimension hidden layer width depth and computes exactly.
Next, consider the general case when is given by
as in (11). For this situation, we use a different way of computing the maximum using nets.
There exists a net with input dimension hidden layer width , output dimension and depth such that
Set and define
We have for each
as desired. ∎
We now describe how to construct a net with input dimension , hidden layer width output dimension and depth that exactly computes . We use width to copy the input , width to compute successive maximums of the positive affine functions using the net from Lemma 4 above, and width as memory in which we store while computing The final layer computes the difference ∎
6. Proof of Theorem 1
We begin by showing (8) and (9). Suppose is convex and fix A simple discretization argument shows that there exists a piecewise affine convex function such that By Theorem 2, can be a exactly represented by a net with hidden layer width This proves (8). In the case that is Lipschitz, we use the following, a special case of Lemma 4.1 in .
Suppose is convex and Lipschitz with Lipschitz constant . Then for every there exist affine maps such that
Let be a net with input dimension a single hidden layer of width and output dimension There exists another net that computes the same function as but has input dimension and hidden layers with width
Denote by the affine functions computed by each neuron in the hidden layer of so that
Let be sufficiently large that
The affine transformations computed by the hidden layer of are then
We are essentially using width to copy in the input variable, width to compute each and width to store the output. ∎
Recall that positive continuous functions can be arbitrarily well-approximated by smooth functions and hence by nets with a single hidden layer (see e.g. Theorem 3.1 ). The relation (5) therefore follows from Lemma 6. Similarly, by Theorem 3.1 in , if is smooth, then there exists and a constant depending only on the maximum value of the first derivatives of such that
of into copies of times the standard -simplex. Define to be a piecewise linear approximation to obtained by setting equal to on the vertices of the ’s and taking to be affine on their interiors. Since the diameter of each is we have
-  R. Arora, A. Basu, P. Mianjy, A. Mukherjee. Understanding deep neural networks with Rectified Linear Units. arXiv:1611.01491v4
-  Y. Bengio, G. Hinton, and Y. LeCun. Deep learning. Nature. Vol. 521, no. 7553, p. 436-444, 2015.
G. Balázs, A, György, and C. Szepesvári.
Near-optimal max-affine estimators for convex regression.
Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics. Vol. 38, p. 56-64, 2015.
Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals, and Systems (MCSS). Vol. 2 no. 4, p. 303-314, 1989.
-  B. Hanin, E. Mossel. Expressivity of nets on the simplex. In preparation.
-  B. Hanin, M. Sellke. Approximating Continuous Functions by ReLU Nets of Minimal Width. arXiv:1710.11278.
K. Hornik, M. Stinchcombe, H. White.
Multilayer feedforward networks are universal approximators
Journal Neural Networks. Vol. 2 no. 5, p. 359-366, 1989.
-  Q. Liao, H. Mhaskar, and T. Poggio. Learning functions: when is deep better than shallow. arXiv:1603.00988v4 (2016).
-  H. Lin, D. Rolnick, M. Tegmark. Why does deep and cheap learning work so well? arXiv:1608.08225v3 (2016).
-  E. Mossel. Mathematical aspects of deep learning: http://elmos.scripts.mit.edu/mathofdeeplearning/mathematical-aspects-of-deep-learning-intro/.
-  H. Mhaskar, T. Poggio. Deep vs. shallow networks: an approximation theory perspective arXiv:1608.03287v1 (2016).
-  B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, S. Ganguli Exponential expressivity in deep neural networks through transient chaos. arXiv:1606.05340 (2016)
-  M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, J. Dickstein. On the expressive power of deep neural nets. arXiv:1606.05336v6 (2017)
-  D. Rolnick, M. Tegmark. The power of deeper networks for expressing natural functions. arXiv:1705.05502v1 (2017)
-  M. Telgrasky. Representation benefits of deep feedforward networks. arXiv:1509.08101 (2015).
-  M. Telgrasky. Benefits of depth in neural nets. JMLR: Workshop and Conference Proceedings vol 49:1–23, 2016.
-  M. Telgrasky. Neural networks and rational functions. arXiv:1706.03301 (2017).
-  D. Yarotsky Error bounds for approximations with deep ReLU network. arXiv:1610.01145 (2017)