
Optimal approximation of continuous functions by very deep ReLU networks
We prove that deep ReLU neural networks with conventional fullyconnecte...
read it

Random ReLU Features: Universality, Approximation, and Composition
We propose random ReLU features models in this work. Its motivation is r...
read it

Deep Equals Shallow for ReLU Networks in Kernel Regimes
Deep networks are often considered to be more expressive than shallow on...
read it

Efficient Approximation of Solutions of Parametric Linear Transport Equations by ReLU DNNs
We demonstrate that deep neural networks with the ReLU activation functi...
read it

Approximation power of random neural networks
This paper investigates the approximation power of three types of random...
read it

Gradient Dynamics of Shallow Univariate ReLU Networks
We present a theoretical and empirical study of the gradient dynamics of...
read it
Neural tangent kernels, transportation mappings, and universal approximation
This paper establishes rates of universal approximation for the shallow neural tangent kernel (NTK): network weights are only allowed microscopic changes from random initialization, which entails that activations are mostly unchanged, and the network is nearly equivalent to its linearization. Concretely, the paper has two main contributions: a generic scheme to approximate functions with the NTK by sampling from transport mappings between the initial weights and their desired values, and the construction of transport mappings via Fourier transforms. Regarding the first contribution, the proof scheme provides another perspective on how the NTK regime arises from rescaling: redundancy in the weights due to resampling allows individual weights to be scaled down. Regarding the second contribution, the most notable transport mapping asserts that roughly 1 / δ^10d nodes are sufficient to approximate continuous functions, where δ depends on the continuity properties of the target function. By contrast, nearly the same proof yields a bound of 1 / δ^2d for shallow ReLU networks; this gap suggests a tantalizing direction for future work, separating shallow ReLU networks and their linearization.
READ FULL TEXT
Comments
There are no comments yet.