Approximation of Smoothness Classes by Deep ReLU Networks

by   Mazen Ali, et al.
Ecole Centrale Nantes

We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces B^α_q(L^p) in arbitrary dimension d, on bounded or unbounded domains. We show that RePU networks with a fixed activation function attain optimal approximation rates for functions in the Besov space B^α_τ(L^τ) on the critical embedding line 1/τ=α/d+1/p for arbitrary smoothness order α>0. Moreover, we show that ReLU networks attain near to optimal rates for any Besov space strictly above the critical line. Using interpolation theory, this implies that the entire range of smoothness classes at or above the critical line is (near to) optimally approximated by deep ReLU/RePU networks.



There are no comments yet.


page 1

page 2

page 3

page 4


Approximation with Tensor Networks. Part III: Multivariate Approximation

We study the approximation of multivariate functions with tensor network...

The phase diagram of approximation rates for deep neural networks

We explore the phase diagram of approximation rates for deep neural netw...

ReLU nets adapt to intrinsic dimensionality beyond the target domain

We study the approximation of two-layer compositions f(x) = g(ϕ(x)) via ...

Two-layer neural networks with values in a Banach space

We study two-layer neural networks whose domain and range are Banach spa...

Optimal Approximation Rates and Metric Entropy of ReLU^k and Cosine Networks

This article addresses several fundamental issues associated with the ap...

How do noise tails impact on deep ReLU networks?

This paper investigates the stability of deep ReLU neural networks for n...

Kernel based analysis of massive data

Dealing with massive data is a challenging task for machine learning. An...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Artificial neural networks (NNs) have become a popular tool in various fields of computational and data science. Due to its popularity and good performance, NNs motivated a lot of research in mathematics – especially in recent years – in an attempt to explain the properties of NNs responsible for their success.

Although many aspects of NNs still lack a satisfactory mathematical explanation, the expressivity or approximation theoretic properties of NNs are by now quite well understood. By expressivity we mean the theoretical capacity of NNs to approximate functions from different classes. We do not intend to give a literature overview on this topic and instead refer to the recent survey in [17].


In this work, we contribute to the existing body of knowledge on the expressivity of NNs by showing that the very popular and yet quite simple feed-forward rectified linear unit (ReLU) NNs can approximate a very wide range of smoothness classes with near to optimal complexity. To make the distinction to existing results clear, we briefly review what is known by now about the approximation of some more standard smoothness classes closely related to our work. In all instances “complexity” is measured by the number of connections, i.e., non-zero weights.

In [23] it was shown that analytic functions on a compact product domain in any dimension can be approximated in the Sobolev norm by ReLU and RePU networks with close to exponential convergence. In [25] it was shown that ReLU networks can approximate any Hölder continuous function with optimal complexity. In [16] it was shown that functions in the Besov space on bounded Lipschitz domains in any dimension can be approximated in the -norm by RePU networks with activation function of degree with optimal complexity. The spaces correspond to the vertical line in Figure 1, i.e., for these are either the same or slightly larger than the Sobolev spaces .

Figure 1. DeVore diagram of smoothness spaces [8]. The Sobolev embedding line is the diagonal with the points and


In [22] it was shown that functions in the Besov space , for on bounded intervals , can be approximated in the -norm111Or in the -norm and consequently, by interpolation, in the fractional Sobolev norm. with near to optimal complexity. The space is above the critical embedding line of functions that barely have enough regularity to be members of , see the diagonal in Figure 1. Spaces above this critical line are embedded in , spaces on this line may or may not be embedded in , and spaces below this line are never embedded in . It was also shown in [22] that piece-wise Gevrey functions can be approximated with close to exponential convergence. Similar results for classical smoothness spaces of univariate functions are contained in [7].

In this work, we show that functions in , for and or a Lipschitz domain in any dimension , can be approximated by RePU networks with activation function of degree with optimal complexity for any . We show the same for ReLU networks with near to222I.e., for any approximation rate arbitrarily close to optimal. optimal complexity and assuming additionally , i.e., for any Besov class strictly above the critical embedding line. This completes the picture for ReLU/RePU expressivity rates for classical smoothness spaces in the sense that, with regard to approximation, functions from any Besov space that embeds into can be approximated by ReLU/RePU networks with (near to) optimal complexity. Note that this feature can be attributed solely to depth as it was observed in [1, 2]

that tensor networks (or sum-product neural networks) – specifically, tensor train networks with a simple nonlinearity – exhibit the same expressivity with regard to classical smoothness spaces.

Upon completion of this work we became aware of a similar result in [26]. We comment on this in detail in Section 5.


We begin in Section 1.1 and Section 1.2 by reviewing the theoretical framework of our work. We then state the main result in Section 1.3. To keep the presentation self-contained, we review previous results on ReLU approximation in Section 2 and smoothness classes in Section 3 that we require for our work. Finally, in Section 4 we derive the main result of this work, stated again in Theorem 4.5. We conclude with a brief discussion on extensions and alternative proofs in Section 5. The reader familiar with results on ReLU/RePU approximation and wavelet characterizations of Besov spaces can skip directly to Section 4.

1.1. Neural Networks

We briefly introduce the mathematical description and notation we use for NNs through out this work. Specifically, we will only consider feed-forward NNs. In Figure 2, we sketch a pictorial representation of a feed-forward NN.

Figure 2.

Example of a feed-forward neural network. On the left we have the

input nodes marked in red that represent input data to the network. The yellow nodes are the neurons that perform some simple operations on the input. The edges between the nodes represent connections that transfer (after possibly applying an affine transformation) the output of one node into the input of another. The final green nodes are the output nodes. In this particular example the number of layers is three, with two hidden layers.

Input values are passed on to the first layer of neurons after possibly undergoing an affine transformation. In the neurons, an activation function is applied to the transformed input values. The result again undergoes an affine transformation and is passed to the next layer and so on, until the output layer is reached.

The number of inputs and outputs is typically determined by the intended application. Specifying the architecture of such an NN amounts to choosing the number of layers, the number of neurons in each hidden layer, the activation functions and the connections or, equivalently, the position of the non-zero weights in the affine transformations. The process of training then consists of determining said weights.

We formalize our description of the considered mathematical objects. Let be the number of layers, the number of inputs, the number of outputs and the number of neurons in each hidden layer. A neural network can be described by the tuple

where for each , is an affine transformation


and is a (nonlinear) function, usually applied component-wise as

In this work we will use RePU activation functions, i.e.,


where is the identity map and for , is referred to as the rectified linear unit (ReLU). We allow for the possibility of a non-strict network, i.e., an activation function is either or . Another possibility is a strict network where each activation function is necessarily (with the exclusion of the output nodes). But, as was shown in [16], the approximation theoretic properties of both are the same and thus, for our work, it is irrelevant.

Let denote the set of affine maps as in (1.1) and denote the set of activation functions as in (1.2). For fixed , , define

and the realization map by

1.2. Approximation Classes

In this work we will state our results in the approximation theoretic framework introduced in [16]. Before we do so, let us first recall the definition of approximation spaces.

Let be a quasi-normed linear space, subsets of for and an approximation tool. Define the best approximation error

With this we define approximation classes as

Definition 1.1 (Approximation Classes).

For any and , define the quantity

The approximation classes of are defined by

The utility of using these classes comes to light only if the sets satisfy certain properties. This was discussed in detail in [16] and the relevant properties were shown to hold for RePU networks.

We perform approximation in for and or a bounded Lipschitz domain. We thus abbreviate

As a measure of complexity we will use the number of non-zero weights. I.e., for a given for some , the number of non-zero weights is

with being the number of non-zero weights of the matrix . With this we define for any

The main result of this work then concerns the approximation classes

1.3. Main Result

For the statement of our main result we will use real -interpolation spaces , see Section 3.1 for a refresher.

Main Result 1.2 (Direct Embeddings).

Let or a Lipschitz domain.

  1. [label=()]

  2. For , and any Besov space with such that

    the following embeddings hold

    for , .

  3. For , and any Besov space with such that

    the following embeddings hold

    for , and any .


For quantities , we will use the notation if there exists a constant that does not depend on or such that . Similarly for and if both inequalities hold. For any , we define

We use to denote the support of a function

and to denote the Lebesgue measure of this set. Finally, we use to denote the standard counting measure.

2. Preliminaries on ReLU Approximation

In this section, we review recent results on deep RePU approximation relevant for this work. We use the notation defined in Section 1.1. The next theorem states that RePU networks can efficiently reproduce or approximate multiplication.

Theorem 2.1 (Multiplication [27, 23, 16]).

Let be the multiplication function . Then, there exists a constant such that

  1. [label=()]

  2. for , and , there exists a RePU network such that

  3. for , any and any , and , there exists a ReLU network with

This in turn implies RePU networks can efficiently reproduce or approximate piece-wise polynomials.

Theorem 2.2 (Piece-wise Polynomials [22]).

Let be a piece-wise polynomial with pieces, of maximum degree and with compact support of measure . Then, there exists a constant depending on , and such that

  1. [label=()]

  2. for , there exists a RePU network with

  3. for , the constant additionally depends on and , and for any there exists a ReLU network with and the same support as , such that

The previous result states that RePU networks with can reproduce piece-wise polynomials of any degree at the same asymptotic cost333Note that the constants will be, however, affected by the degree.. This suggests the following saturation property.

Theorem 2.3 (Saturation Property [16]).

For any , any and , and any the approximation spaces defined in Section 1.2 coincide

The saturation property will also be clearly visible in the main result of this work in Theorem 4.5.

We conclude by pointing out that RePU networks can efficiently reproduce affine systems, i.e., linear combinations of functions that are generated by dilating and shifting a single mother function or, in some cases, a finite number of mother functions. A prominent example of affine systems are wavelets which will play an important role for the main result in Theorem 4.5.

The reproduction of affine systems by NNs was studied in greater detail in [3]. In the following we only mention the properties relevant for this work.

Theorem 2.4 (NN calculus [16]).

For any , the following properties hold.

  1. [label=()]

  2. For any , , and any , there exists with

  3. For any , and any , set . Then, for , there exists with

  4. For any , and any , set and . Then, for , there exists with

  5. For any , , any and any , there exists such that

  6. Let denote the affine transformation for , . Then, for any , , any and any , , there exists with

3. Besov Spaces and Wavelet Systems

In this section, we recall some classical results on (isotropic) Besov spaces and their characterization with wavelets. As in Section 2, we focus mostly on results relevant to our work. For more details we refer to, e.g., [4].

3.1. Besov Spaces

Let be an open subset and for . For , let denote the translation operator , the identity operator and define the -th difference

We use the notation

The modulus of smoothness of order is defined for any as

where denotes the standard Euclidean -norm. Finally, the Besov semi-norm is defined for any , any and by

Then, the (isotropic) Besov space is defined as

and it is a (quasi-)Banach space equipped with the norm

The parameter is the smoothness order, while the primary parameter reflects the measure of said smoothness. The secondary parameter is less important and merely provides a finer gradation of smoothness. A few relationships are rather straight-forward

where denotes a continuous embedding. For non-integer and , is the fractional Sobolev space . For integer , the Besov space is slightly larger than . For , the Besov space is the same as the Sobolev space .

A less obvious property are the following embedding results.

Theorem 3.1 (Besov Embeddings [4]).

Let or be a Lipschitz domain.

  1. [label=()]

  2. For , , and such that , the following embedding holds

  3. For , and , it holds

The Besov spaces in Theorem 3.1 (i) are on the critical embedding line (see Figure 1). Spaces above this line are embedded in , spaces on this line may or may not be embedded in , and spaces below this line are never embedded in . In this sense, such Besov spaces are quite large as the functions on this line barely have enough regularity to be members of . It is well-known that optimal approximation of functions from such spaces with a continuous parameter selection can only be achieved by non-linear methods, see [9]. It is the main result of this work that RePU networks achieve optimal approximation for these spaces, while ReLU networks achieve near to optimal approximation.

To transfer results from to bounded Lipschitz domains, we will use the common technique of extension operators.

Theorem 3.2 (Extension Operator [19, 13]).

Let be a Lipschitz domain444The result is actually valid for more general domains (cf.  domains), see [13].. Then, for any and any , there exists a linear operator such that

where depends only on , , and the domain .

We conclude by noting that Besov spaces combine well with interpolation. To be precise, we briefly define interpolation spaces via the -functional. Let be a quasi-normed space and be a quasi-semi-normed space with . The -functional is defined for any by

For and , define the quantity

Then, the spaces

equipped with the (quasi-)norm

are interpolation spaces.

Besov spaces provide a relatively complete description of interpolation spaces in the following sense: for

For Besov spaces on the critical line with , we obtain

3.2. Wavelets

There are many possible wavelets constructions satisfying different properties depending on the intended application. Said constructions can be rather technical, with the payoff being various favorable analytical and numerical features. We do not intend to cover this topic in-depth and once again only pick out the aspects required for this work. We proceed by briefly reviewing one-dimensional wavelets constructions, after which we turn to wavelets on . Our presentation is somewhat abstract and therefore flexible, but we will also be more specific with some aspects of the construction that we require in Section 4. For more details on the subject we refer to [4].

The starting point of a wavelet construction is typically a multi-resolution analysis (MRA), i.e., a sequence of closed subspaces of that are nested, dilation- and shift-invariant, dense in and are all generated by a single555Multiple scaling functions are possible as well in which case such functions are referred to as multi-wavelets, see [14]. scaling function . To be more precise, we assume the system is a Riesz basis of and therefore is a Riesz basis of . We use the shorthand notation

where the pre-factor normalizes in . Later we will redefine this to for normalization in for any , with the convention .

Defining a projection is rather simple if forms an orthogonal basis of . Indeed, this property implies that forms an orthogonal basis of , and can be chosen to be the orthogonal projection. However, for numerical reasons, it is sometimes unpractical to construct scaling functions such that forms an orthogonal basis of and without this property a constructive definition of is not straight-forward.

A way-out are so-called bi-orthogonal constructions. A function is dual to if it satisfies

where is the Kronecker delta. We then define the oblique projection


A representation of a function in is typically referred to as a single-scale representation. To switch to a multi-scale representation, we need to characterize the so-called detail spaces defined through the projections

with the detail spaces defined as . This is achieved by constructing a wavelet

for some coefficients such that


Any function can then be decomposed into a sequence of single-scale coefficients on the coarsest level and detail coefficients on all higher levels


To simplify notation, one typically sets and introduces the index set . Decomposition (3.3) then simplifies to

In order for the wavelets to characterize Besov spaces, they have to satisfy certain assumptions.

Assumption 3.3 (Characterization).

We assume the scaling function and its dual satisfy the following properties.

  1. [label=(W0)]

  2. (Integrability) For some such that , we assume and .

  3. (Polynomial Reproduction) We assume satisfies Strang-Fix conditions of order or, equivalently, for any polynomial of degree , we have .

  4. (Regularity) For some , , we assume .

These conditions are sufficient to ensure Besov spaces can be characterized by the decay of the wavelet coefficients. For our work we will require two additional conditions that are, however, easy to satisfy for a variety of wavelet families.

Assumption 3.4 (Piece-wise Polynomial).

We additionally assume the scaling function satisfies the following properties.

  1. [label=(A0)]

  2. We assume has compact support.

  3. We assume is piece-wise polynomial.

An example of a wavelet family satisfying all of the assumptions 13 and 12 are the CDF bi-orthogonal B-spline wavelets from [6]. These constructions allow to choose an arbitrary polynomial reproduction degree , regularity order and the resulting scaling function (and consequently as well) are compactly supported splines of degree .

Finally, we briefly describe how to extend the above wavelets to . There are several possible approaches for this, but we describe a specific tensor product construction suitable for isotropic Besov spaces. We comment on anisotropic Besov spaces in Section 5.

For , we define the tensor product scaling function as


and in the same manner as before, but for a general ,


with the convention . Next, for , we define


with the convention and , and is defined as in (3.5). Simplifying as before with

we obtain the -dimensional wavelet system


where we also use the shorthand notation for , and otherwise. Finally, we define the fixed level sets

Theorem 3.5 (Characterization [4]).

Let satisfy 1 for some integrability parameters , 2 for order and 3 with smoothness order for primary parameter and any secondary parameter . Then, if is the wavelet decomposition of , for


we have the norm equivalence

The above characterization implies optimal approximation rates for best -term wavelet approximations.

Theorem 3.6 (-term Approximation [4]).

Let and let be a wavelet system satisfying the assumptions of Theorem 3.5. Define the set of -term wavelet expansions as

Then, for , and any , it holds

Remark 3.7.

For the corresponding Besov space is the space of Hölder continuous functions and we refer to [11, 25]. The restriction stems from (3.8) which in turn is based on the fact that the oblique projector defined in (3.1) is, in general, not -stable for . More on this in Section 5.

4. Optimal ReLU Approximation of Smoothness Classes

With the results from Section 2 and Section 3 we have all the tools necessary to derive approximation rates for arbitrary Besov functions. As was reviewed in Section 3, Besov spaces can be characterized by the decay of the wavelet coefficients, and -term approximations achieve optimal approximation rates for Besov functions.

In this section, we show that a RePU network666Of bounded depth depending on the smoothness order and polynomial degree of the activation function. can reproduce an -term wavelet expansion with complexity. More importantly, we also show that a ReLU network777Of depth depending logarithmically on . can approximate an -term wavelet expansion with