The universal approximation power of finite-width deep ReLU networks

06/05/2018
by   Dmytro Perekrestenko, et al.
1

We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with their recently established universal approximation property of affine function systems (Bölcskei et al., 2018), this shows that deep neural networks approximate vastly different signal structures generated by the affine group, the Weyl-Heisenberg group, or through warping, and even certain fractals, all with approximation error decaying exponentially in the number of neurons. We also prove that in the approximation of sufficiently smooth functions finite-width deep networks require strictly smaller connectivity than finite-depth wide networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/09/2020

Deep Network Approximation for Smooth Functions

This paper establishes optimal approximation error characterization of d...
research
02/28/2019

A lattice-based approach to the expressivity of deep ReLU neural networks

We present new families of continuous piecewise linear (CPWL) functions ...
research
01/08/2019

Deep Neural Network Approximation Theory

Deep neural networks have become state-of-the-art technology for a wide ...
research
10/07/2021

Tighter Sparse Approximation Bounds for ReLU Neural Networks

A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski Ba...
research
06/13/2019

Neural Networks on Groups

Recent work on neural networks has shown that allowing them to build int...
research
02/01/2020

A Corrective View of Neural Networks: Representation, Memorization and Learning

We develop a corrective mechanism for neural network approximation: the ...
research
06/22/2018

On the Spectral Bias of Deep Neural Networks

It is well known that over-parametrized deep neural networks (DNNs) are ...

Please sign up or login with your details

Forgot password? Click here to reset