Minimum complexity interpolation in random features models

03/30/2021
by   Michael Celentano, et al.
0

Despite their many appealing properties, kernel methods are heavily affected by the curse of dimensionality. For instance, in the case of inner product kernels in ℝ^d, the Reproducing Kernel Hilbert Space (RKHS) norm is often very large for functions that depend strongly on a small subset of directions (ridge functions). Correspondingly, such functions are difficult to learn using kernel methods. This observation has motivated the study of generalizations of kernel methods, whereby the RKHS norm – which is equivalent to a weighted ℓ_2 norm – is replaced by a weighted functional ℓ_p norm, which we refer to as ℱ_p norm. Unfortunately, tractability of these approaches is unclear. The kernel trick is not available and minimizing these norms requires to solve an infinite-dimensional convex problem. We study random features approximations to these norms and show that, for p>1, the number of random features required to approximate the original learning problem is upper bounded by a polynomial in the sample size. Hence, learning with ℱ_p norms is tractable in these cases. We introduce a proof technique based on uniform concentration in the dual, which can be of broader interest in the study of overparametrized models.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/27/2019

On the Risk of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels

We study the risk of minimum-norm interpolants of data in a Reproducing ...
08/07/2020

Generalization error of minimum weighted norm and kernel interpolation

We study the generalization error of functions that interpolate prescrib...
05/31/2011

Approximation properties of certain operator-induced norms on Hilbert spaces

We consider a class of operator-induced norms, acting as finite-dimensio...
06/02/2020

Non-asymptotic Analysis in Kernel Ridge Regression

We develop a general non-asymptotic analysis of learning rates in kernel...
09/09/2008

Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning

For supervised and unsupervised learning, positive definite kernels allo...
01/22/2021

On the robustness of certain norms

We study a family of norms defined for functions on an interval. These n...
04/24/2019

Native Banach spaces for splines and variational inverse problems

We propose a systematic construction of native Banach spaces for general...