
A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
A key element of understanding the efficacy of overparameterized neural ...
read it

Approximating Continuous Functions by ReLU Nets of Minimal Width
This article concerns the expressive power of depth in deep feedforward...
read it

Inductive Bias of MultiChannel Linear Convolutional Networks with Bounded Weight Norm
We study the function space characterization of the inductive bias resul...
read it

Fundamental tradeoffs between memorization and robustness in random features and neural tangent regimes
This work studies the (non)robustness of twolayer neural networks in va...
read it

Examples, counterexamples, and structure in bounded width algebras
We study bounded width algebras which are minimal in the sense that ever...
read it

Entropy versus influence for complex functions of modulus one
We present an example of a function f from {1,1}^n to the unit sphere i...
read it

Convergence of the conjugate gradient method with unbounded operators
In the framework of inverse linear problems on infinitedimensional Hilb...
read it
How do infinite width bounded norm networks look in function space?
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal norm required to approximate a given function. For functions f : R → R and a single hidden layer, we show that the minimal network norm for representing f is (∫ f"(x) dx, f'(∞) + f'(+∞)), and hence the minimal norm fit for a sample is given by a linear spline interpolation.
READ FULL TEXT
Comments
There are no comments yet.