Identification of Shallow Neural Networks by Fewest Samples

04/04/2018
by   Massimo Fornasier, et al.
0

We address the uniform approximation of sums of ridge functions ∑_i=1^m g_i(a_i· x) on R^d, representing the shallowest form of feed-forward neural network, from a small number of query samples, under mild smoothness assumptions on the functions g_i's and near-orthogonality of the ridge directions a_i's. The sample points are randomly generated and are universal, in the sense that the sampled queries on those points will allow the proposed recovery algorithms to perform a uniform approximation of any sum of ridge functions with high-probability. Our general approximation strategy is developed as a sequence of algorithms to perform individual sub-tasks. We first approximate the span of the ridge directions. Then we use a straightforward substitution, which reduces the dimensionality of the problem from d to m. The core of the construction is then the approximation of ridge directions expressed in terms of rank-1 matrices a_i ⊗ a_i, realized by formulating their individual identification as a suitable nonlinear program, maximizing the spectral norm of certain competitors constrained over the unit Frobenius sphere. The final step is then to approximate the functions g_1,...,g_m by ĝ_1,...,ĝ_m. Higher order differentiation, as used in our construction, of sums of ridge functions or of their compositions, as in deeper neural network, yields a natural connection between neural network weight identification and tensor product decomposition identification. In the case of the shallowest feed-forward neural network, we show that second order differentiation and tensors of order two (i.e., matrices) suffice.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/26/2016

Uniform Approximation by Neural Networks Activated by First and Second Order Ridge Splines

We establish sup-norm error bounds for functions that are approximated b...
research
06/30/2020

Approximation with Tensor Networks. Part I: Approximation Spaces

We study the approximation of functions by tensor networks (TNs). We sho...
research
02/25/2021

Recovery of regular ridge functions on the ball

We consider the problem of the uniform (in L_∞) recovery of ridge functi...
research
06/09/2022

A general approximation lower bound in L^p norm, with applications to feed-forward neural networks

We study the fundamental limits to the expressive power of neural networ...
research
06/30/2019

Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks

We address the structure identification and the uniform approximation of...
research
12/22/2020

Improving Sample and Feature Selection with Principal Covariates Regression

Selecting the most relevant features and samples out of a large set of c...
research
09/14/2021

Approximation of Curve-based Sleeve Functions in High Dimensions

Sleeve functions are generalizations of the well-established ridge funct...

Please sign up or login with your details

Forgot password? Click here to reset