Neural networks are learning machines motivated by the architecture of the human brain. Neural networks are comprised of multiple hidden layers, and each of the hidden layers has multiple hidden nodes which consist of an affine map of the outputs from the previous layer and a nonlinear map called an activation function. Deep neural networks have been leading tremendous success in various pattern recognition and machine learning tasks such as object recognition, image segmentation, machine translation and others. For an overview on the empirical success of deep neural networks, we refer to the review paper and recent book .
Inspired by the success of deep neural networks, many researchers have tried to give theoretical supports for the success of deep neural networks. Much of the work upto date has focused on the expressivity of deep neural networks, i.e., ability to approximate a rich class of functions efficiently. The well-known classical result on this topic is the universal approximation theorem, which states that every continuous function can be approximated arbitrarily well by a neural network [11, 15, 12, 5, 20]. But these results do not specify the required numbers of layers and nodes of a neural network to achieve a given approximation accuracy.
Recently, several results about the effects of the numbers of layers and nodes of a deep neural network to its expressivity have been reported. They provide upper bounds of the numbers of layers and nodes required for neural networks to uniformly approximate all functions of interest. Examples of a class of functions include the space of rational functions of polynomials , the Hölder space [33, 27, 2, 21], Besov and mixed Besov spaces  and even a class of discontinuous functions [25, 16].
The nonlinear activation function is a central part that makes neural networks differ from the linear models, that is, a neural network becomes a linear function if the linear activation function is used. Therefore, the choice of an activation function substantially influences on the performance and computational efficiency. Numerous activation functions have been suggested to improve neural network learning [3, 6, 4, 26, 18, 32]. We refer to the papers [13, 26] for an overview of this topic.
There are also many recent theoretical studies about the approximation ability of deep neural networks. However, most of the studies focus on a specific type of the activation function such as ReLU [33, 27, 25, 16, 29]
, or small classes of activation functions such as sigmoidal functions with additional monotonicity, continuity, and/or boundedness conditions[24, 9, 8, 10, 7] and -admissible functions which are sufficiently smooth and bounded . For definitions of sigmoidal and -admissible functions, see  and , respectively. Thus a unified theoretical framework still lacks.
In this paper, we investigate the approximation ability of deep neural networks with a quite general class of activation functions. We derive the required numbers of layers and nodes of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Our specified class of activation functions and the corresponding approximation ability of deep neural networks include most of previous results [33, 27, 24, 2] as special cases.
Our general theoretical results of the approximation ability of deep neural networks enables us to study statistical properties of deep neural networks. Schmidt-Hieber  and Kim et al.  proved the minimax optimality of a deep neural network estimator with the ReLU activation function in regression and classification problems, respectively. In this paper, we derive similar results for general activation functions.
This paper is structured as follows. In Section 2, we introduce some notions about deep neural networks. In Section 3, we introduce two large classes of activation functions. In Section 4, we present our main result on the approximation ability of a deep neural network with the general activation function considered in Section 3. In Section 5, we apply the result in Section 4
to the supervised learning problems of regression and classification. Conclusions are given in Section6. The proofs of all results are given in Appendix.
We denote by the indicator function. Let be the set of real numbers and
be the set of natural numbers. For a real valued vector, we let , for and . For simplicity, we let For a real valued function , we let and are the first, second and third order derivatives of at , respectively. We let and . For , we write .
2 Deep Neural Networks
In this section we provide a mathematical representation of deep neural networks. A neural network with layers, many nodes at the -th hidden layer for , input of dimension , output of dimension and nonlinear activation function is expressed as
where is an affine linear map defined by for given dimensional weight matrix and
dimensional bias vectorand is an element-wise nonlinear activation map defined by . Here, denotes the set of all weight matrices and bias vectors which we call the parameter of the neural network, or simply, a network parameter.
We introduce some notations related to the network parameter. For a network parameter , we write for the number of hidden layers of the corresponding neural network, and write for the maximum of the numbers of hidden nodes at each layer. Following a standard convention, we say that is the depth of the deep neural network and is the width of the deep neural network. We let be the number of nonzero elements of , i.e.,
where transforms the matrix into the corresponding vector by concatenating the column vectors. We call sparsity of the deep neural network. Let be the largest absolute value of elements of , i.e.,
We call magnitude of the deep neural network. We let and be the input and output dimensions of the deep neural network, respectively. We denote by the set of network parameters with depth , width , input dimension and output dimension , that is,
We further define a subset of with restrictions on sparsity and magnitude as
3 Classes of Activation Functions
In this section, we consider two classes of activation functions. These two classes include most of commonly used activation functions. Definitions and examples of each class of activation functions are provided in the consecutive two subsections.
3.1 Piecewise Linear Activation Functions
We first consider piecewise linear activation functions.
A function is continuous piecewise linear if it is continuous and there exist a finite number of break points with such that for every and is linear on .
Throughout this paper, we write “picewise linear” instead of “continuous picewise linear” for notational simplicity unless there is a confusion. The representative examples of piecewise linear activation functions are as follows:
Leaky ReLU: : for .
The ReLU activation function is the most popular choice in practical applications due to better gradient propagation and efficient computation . In this reason, most of the recent results on the function approximation by deep neural networks are based on the ReLU activation function [33, 27, 25, 16, 29]. In Section 4, as Yarotsky  did, we extend these results to any continuous piecewise linear activation function by showing that the ReLU activation function can be exactly represented by a linear combination of piecewise linear activation functions. A formal proof for this argument is presented in Appendix A.1.
3.2 Locally Quadratic Activation Functions
One of the basic building blocks in approximation by deep neural networks is the square function, which should be approximated precisely. Piecewise linear activation functions have zero curvature (i.e., constant first-order derivative) inside each interval divided by its break points, which makes it relatively difficult to approximate the square function efficiently. But if there is an interval on which the activation function has nonzero curvature, the square function can be approximated more efficiently, which is a main motivation of considering a new class of activation functions that we call locally quadratic.
A function is locally quadratic if there exits an open interval on which is three times continuously differentiable with bounded derivatives and there exists such that and .
We now give examples of locally quadratic activation functions. First of all, any nonlinear smooth activation function with nonzero second derivative, is locally quadratic. Examples are:
In addition, piecewise smooth function having nonzero second derivative on at least one piece, is also locally quadratic. Examples are:
4 Approximation of Smooth Functions by Deep Neural Networks
In this section we introduce the function class we consider and show the approximation ability of the deep neural networks with a activation function considered in Section 3.
4.1 Hölder Smooth Functions
We recall the definition of Hölder smooth functions. For a -dimensional multiple index where , we let for . For a function , where denotes the domain of the function, we let . We use notation
for to denote a derivative of of order . We denote by , the space of times differentiable functions on whose partial derivatives of order with are continuous. We define the Hölder coefficient of order as
For a positive real value , the Hölder space of order is defined as
where denotes the Hölder norm defined by
We denote by the closed ball in the Hölder space of radius with respect to the Hölder norm, i.e.,
4.2 Approximation of Hölder Smooth Functions
We present our main theorem in this section.
Let , and . Let the activation function be either continuous piecewise linear or locally quadratic. Let . Then there exist positive constants , , and depending only on , , and such that, for any , there is a neural network
The result of Theorem 4.1 is equivalent to the results on the approximation by ReLU neural networks [33, 27] in a sense that the upper bounds of the depth, width and sparsity are the same orders of those for ReLU, namely, depth , width and sparsity . We remark that each upper bound is equivalent to the corresponding lower bound established by  up to logarithmic factor.
For piecewise linear activation functions, Yarotsky  derived similar results to ours. For locally quadratic activation functions, only special classes of activation functions were considered in the previous work. Li et al.  considered the RePU activation function and Bauer and Kohler  considered sufficiently smooth and bounded activation functions which include the sigmoid, tangent hyperbolic, ISRU and soft clipping activation functions. However, soft plus, swish, ELU, ISRLU, softsign and square nonlinearity activation functions are new ones only considered in our results.
Even if the orders of the depth, width and sparsity are the same for both both piecewise linear and locally quadratic activation functions, the ways of approximating a smooth function by use of these two activation function classes are quite different. To describe this point, let us provide an outline of the proof. We first consider equally spaced grid points with length inside the -dimensional unit hypercube . Let be the set of such grid points, namely,
For a given Hölder smooth function of order , we first find a “local” function for each grid that approximates the target function near the grid point but vanishes at apart from the grid point. To be more specific, we construct the local functions , which satisfies:
for some universal constant . The inequality (4.3) implies that the more grid points we used, the more accurate approximation we get. Moreover, the quality of approximation is improved when the target function is more smooth (i.e., large ) and low dimensional (i.e., small ). In fact, is given by a product of the Taylor polynomial at and the local basis function , where . By simple algebra, we have
The second stage is to approximate each monomial and each local basis function by deep neural networks. Each monomial can be approximated more efficiently by a deep neural network with a locally quadratic activation function than a piecewise linear activation function since each monomial has nonzero curvature. On the other hand, the local basis function can be approximated more efficiently by a deep neural network with a piecewise linear activation than a locally quadratic activation function since the local basis function is piecewise linear itself. That is, there is a trade-off in using either a piecewise linear or a locally quadratic activation function.
We close this section by giving a comparison of our result to the approximation error analysis of . Bauer and Kohler  studies approximation of the Hölder smooth function of order by a two layer neural network with -admissible activation functions with , where a function is called -admissible if (1) is at least times continuously differentiable with bounded derivatives; (2) a point exists, where all derivatives up to the order of are different from zero; and (3) for and for . Our notion of locally quadratic activation functions is a generalized version of the -admissibility.
In the proof of , the condition is necessary because they approximate any monomial of order with with a two layer neural network, which is impossible when . We drop the condition by showing that any monomial of order with can be approximated by deep neural network with a finite number of layers, which depends on .
5 Application to Statistical Learning Theory
In this section, we apply our results about the approximation error of neural networks to the supervised learning problems of regression and classification. Let be the input space and the output space. Let be a given class of measurable functions from to . Let be the true but unknown data generating distribution on . The aim of supervised learning is to find a predictive function that minimizes the population risk
with respect to a given loss function. Since is unknown, we cannot directly minimize the population risk, and thus any estimator inevitably has the excess risk which is defined as For a given sample of size , let be a given subset of called a sieve and let be observed (training) data of input–output pairs assumed to be independent realizations of following Let be an estimated function among based on the training data The excess risk of is decomposed to approximation and estimation errors as
There is a trade-off between approximation and estimation errors. If the function class is sufficiently large to approximate the optimal estimator
well, then the estimation error becomes large due to high variance. In contrast, ifis small, it leads to low estimation error but it suffers from large approximation error.
One of the advantages of deep neural networks is that we can construct a sieve which has good approximation ability as well as low complexity. Schmidt-Hieber  and Kim et al.  proved that a neural network estimator can achieve the optimal balance between the approximation and estimation errors to obtain the minimax optimal convergence rates in regression and classification problems, respectively. But they only considered the ReLU activation function. Based on the results of Theorem 4.1, we can easily extend their results to general activation functions.
The main tool to derive the minimax optimal convergence rate is that the complexity of a class of functions generated by a deep neural network is not affected much by a choice of an activation function, provided that the activation function is Lipschitz continuous. The function is Lipschitz continuous if there is a constant such that
for any . Here, is called the Lipschitz constant. We use the covering number with respect to the norm as a measure of complexity of function classes. We recall the definition of the covering number. Let be a given class of real-valued functions defined on . Let . A collection is called a -covering set of with respect to the norm if for all , there exists in the collection such that . The cardinality of the minimal -covering set is called the -covering number of with respect to the norm which is denoted by That is,
where . The following proposition provides the covering number of a class of functions generated by neural networks.
Assume that the activation function is Lipschitz continuous with the Lipschitz constant . Consider a class of functions generated by a deep neural network
For any ,
The result in Proposition 5.1 is very similar to the existing results in literature, e.g., Theorem 14.5 of , Lemma 5 of  and Lemma 3 of . We employ similar techniques used in [1, 27, 29] to obtain the version presented here. We give the proof of this proposition in Appendix B.
All of the activation functions considered in Section 3 except RePU satisfy the Lipschitz condition (5.2) and hence Proposition 5.1 can be applied. An interesting implication of Proposition 5.1 is that the complexity of the function class generated by deep neural networks is not affected by the choice of an activation function. Hence, the remaining step to derive the convergence rate of a neural network estimator is that approximation accuracies by various activation functions are the same as that of the ReLU neural network.
5.1 Application to Regression
First we consider the regression problem. For simplicity, we let Suppose that the generated model is for some The performance of an estimator is measured by the risk , which is defined by
where is the marginal distribution of The following theorem proves that the optimal convergence rate is obtained by the deep neural network estimator of the regression function for a general activation function.
Suppose that the activation function is either piecewise linear or locally quadratic satisfying the Lipschitz condition (5.2). Then there are universal positive constants , , and such that the deep neural network estimator obtained by
for some satisfies
for some universal constant , where the expectation is taken over the training data.
5.2 Application to Binary Classification
The aim of the binary classification is to find a classifier that predicts the labelfor any input . An usual assumption on the data generating process is that for some , where
denotes the Bernoulli distribution with parameter. Note that
is the conditional probability functionA common approach is, instead of finding a classifier directly, to construct a real valued function , a so-called classification function, and predict the label based on the sign of . The performance of a classification function is measured by the misclassification error , which is defined by
It is well known that the convergence rate of the excess risk for classification is faster than that of regression when the conditional probability function satisfies the following condition: there is a constant such that for any sufficiently small , we have
This condition is called the Tsybakov noise condition and is called the noise exponent [23, 31]. When is larger, the classification task is easier since the probability of generating vague samples become smaller. The following theorem proves that the optimal convergence rate can be obtained by the deep neural network estimator with an activation function considered in Section 3. As is done by , we consider the hinge loss
Assume the Tsybakov noise condition (5.4) with the noise exponent . Suppose that the activation function , which is either piecewise linear or locally quadratic satisfying the Lipschitz condition (5.2), is used for all hidden layers except the last one and the ReLU activation function is used for the last hidden layer. Then there are universal positive constants , , and such that the deep neural network estimator obtained by
for and some satisfies
for some universal constant , where the expectation is taken over the training data.
Note that the Bayes classifier is given by
which is an indicator function. Since a neural network with the ReLU activation function can approximate indicator functions well [25, 16, 17], we use the ReLU activation function in the last layer in order to approximate the Bayes classifier more precisely and thus to achieve the optimal convergence rate.
In this study, we established the upper bounds of the required depth, width and sparsity of deep neural networks to approximate any Hölder continuous function for the general classes of activation functions. These classes of activation functions include most of the popularly used activation functions. The derived upper bounds of the depth, width and sparsity are optimal in a sense that they are equivalent to the lower bounds up to logarithmic factors. We used this generalization of the approximation error analysis to extend the statistical optimality of the deep neural network estimator in regression and classification problems, where the activation function is other than the ReLU.
Our construction of neural networks for approximation reveals that the piecewise linear activation functions are more efficient in approximating local basis functions while locally quadratic activation functions are more efficient in approximating polynomials. Hence if the activation function has both piecewise linear region and locally quadratic region, we could have a better approximation result. We leave the development of such activation functions as a future work.
Appendix A Proof of Theorem 4.1
a.1 Proof of Theorem 4.1 for Piecewise Linear Activation Functions
The main idea of the proof is that any deep neural network with the ReLU activation function can be exactly reconstructed by a neural network with a piecewise activation function whose proof is in the next lemma that is a slight modification of Proposition 1 (b) of .
Let be an any continuous peicewise linear activation function, and be the ReLU activation function. Let . Then there exists such that
where is a constant depending on the activation function .
Let be any break point of . Note that . Let be the distance between and the closest other break point. Then is linear on and . Then for any , the ReLU activation function is expressed as
for any , where we define , and .
Let be given. Since both input and the network parameter are bounded, we can take a sufficiently large so that Equation (A.1) holds for any hidden nodes of the network . We consider the deep neural network , where we set
Here, denotes the -dimensional vector of . Then by Equation (A.1) and some algebra, we have that for any . For the sparsity of , we note that
which implies that . ∎
Thanks to Lemma A.1, to prove Theorem 4.1 for piecewise linear activation functions, it suffices to show the approximation ability of the ReLU networks, which is already done by  as in the next lemma.
Lemma A.2 (Theorem 5 of ).
Let be the ReLU activation function. For any and any integers and , there exists a network parameter such that
where , , and .
Proof of Theorem 4.1 for piecewise linear activation functions.
Let be the ReLU activation function. By letting and Lemma A.2 implies that there exists a network parameter such that with , and for some positive constants , , and depending only on , and . Hence by Lemma A.1, there is a network parameter producing the same output of the ReLU neural network with , , and for some depending only on , , and , and some depending only on , which completes the proof. ∎
a.2 Proof of Theorem 4.1 for Locally Quadratic Activation Functions
Assume that an activation function is locally quadratic. There is a constant depending only on the activation function such that for any the following results hold.
There is a neural network with such that
where is a constant depending only on .
Let . There is a neural network parameter with such that
Let be a positive integer. For any multi-index with , there is a network parameter with such that