The necessity of depth for artificial neural networks to approximate certain classes of smooth and bounded functions without the curse of dimensionality

01/19/2023
by   Lukas Gonon, et al.
0

In this article we study high-dimensional approximation capacities of shallow and deep artificial neural networks (ANNs) with the rectified linear unit (ReLU) activation. In particular, it is a key contribution of this work to reveal that for all a,b∈ℝ with b-a≥ 7 we have that the functions [a,b]^d∋ x=(x_1,…,x_d)↦∏_i=1^d x_i∈ℝ for d∈ℕ as well as the functions [a,b]^d∋ x =(x_1,…, x_d)↦sin(∏_i=1^d x_i) ∈ℝ for d ∈ℕ can neither be approximated without the curse of dimensionality by means of shallow ANNs nor insufficiently deep ANNs with ReLU activation but can be approximated without the curse of dimensionality by sufficiently deep ANNs with ReLU activation. We show that the product functions and the sine of the product functions are polynomially tractable approximation problems among the approximating class of deep ReLU ANNs with the number of hidden layers being allowed to grow in the dimension d ∈ℕ. We establish the above outlined statements not only for the product functions and the sine of the product functions but also for other classes of target functions, in particular, for classes of uniformly globally bounded C^∞-functions with compact support on any [a,b]^d with a∈ℝ, b∈(a,∞). Roughly speaking, in this work we lay open that simple approximation problems such as approximating the sine or cosine of products cannot be solved in standard implementation frameworks by shallow or insufficiently deep ANNs with ReLU activation in polynomial time, but can be approximated by sufficiently deep ReLU ANNs with the number of parameters growing at most polynomially.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/12/2023

Deep neural network approximation of composite functions without the curse of dimensionality

In this article we identify a general class of high-dimensional continuo...
research
07/24/2023

Rates of Approximation by ReLU Shallow Neural Networks

Neural networks activated by the rectified linear unit (ReLU) play a cen...
research
07/28/2021

Neural Network Approximation of Refinable Functions

In the desire to quantify the success of neural networks in deep learnin...
research
09/02/2022

Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality

In this note, we study how neural networks with a single hidden layer an...
research
12/18/2021

The Kolmogorov Superposition Theorem can Break the Curse of Dimensionality When Approximating High Dimensional Functions

We explain how to use Kolmogorov's Superposition Theorem (KST) to overco...
research
03/06/2023

On the existence of optimal shallow feedforward networks with ReLU activation

We prove existence of global minima in the loss landscape for the approx...

Please sign up or login with your details

Forgot password? Click here to reset