Minimum complexity interpolation in random features models

by   Michael Celentano, et al.

Despite their many appealing properties, kernel methods are heavily affected by the curse of dimensionality. For instance, in the case of inner product kernels in ℝ^d, the Reproducing Kernel Hilbert Space (RKHS) norm is often very large for functions that depend strongly on a small subset of directions (ridge functions). Correspondingly, such functions are difficult to learn using kernel methods. This observation has motivated the study of generalizations of kernel methods, whereby the RKHS norm – which is equivalent to a weighted ℓ_2 norm – is replaced by a weighted functional ℓ_p norm, which we refer to as ℱ_p norm. Unfortunately, tractability of these approaches is unclear. The kernel trick is not available and minimizing these norms requires to solve an infinite-dimensional convex problem. We study random features approximations to these norms and show that, for p>1, the number of random features required to approximate the original learning problem is upper bounded by a polynomial in the sample size. Hence, learning with ℱ_p norms is tractable in these cases. We introduce a proof technique based on uniform concentration in the dual, which can be of broader interest in the study of overparametrized models.


page 1

page 2

page 3

page 4


On the Risk of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels

We study the risk of minimum-norm interpolants of data in a Reproducing ...

Generalization error of minimum weighted norm and kernel interpolation

We study the generalization error of functions that interpolate prescrib...

Approximation properties of certain operator-induced norms on Hilbert spaces

We consider a class of operator-induced norms, acting as finite-dimensio...

Non-asymptotic Analysis in Kernel Ridge Regression

We develop a general non-asymptotic analysis of learning rates in kernel...

Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning

For supervised and unsupervised learning, positive definite kernels allo...

On the robustness of certain norms

We study a family of norms defined for functions on an interval. These n...

Native Banach spaces for splines and variational inverse problems

We propose a systematic construction of native Banach spaces for general...