Statistical Mechanics of Generalization in Kernel Regression

by   Abdulkadir Canatar, et al.

Generalization beyond a training dataset is a main goal of machine learning. We investigate generalization error in kernel regression using statistical mechanics, deriving an analytical expression applicable to any kernel. We discuss applications to a kernel with finite number of spectral modes. Then, focusing on the broad class of rotation invariant kernels, which is relevant to training deep neural networks in the infinite-width limit, we show several phenomena. When data is drawn from a spherically symmetric distribution and the number of input dimensions, D, is large, we find that multiple learning stages exist, one for each scaling of the number of training samples with 𝒪_D(D^K) where K∈ Z^+. The behavior of the learning curve in each stage is related to an effective noise and regularizer that are related to the tail of the kernel and target function spectra. When effective regularization is zero, we identify a first order phase transition that corresponds to a divergence in the generalization error. Each learning stage can exhibit sample-wise double descent, where learning curves show non-monotonic sample size dependence. For each stage an optimal value of effective regularizer exists, equal to the effective noise variance, that gives minimum generalization error.


page 1

page 2

page 3

page 4

∙ 06/04/2021

Out-of-Distribution Generalization in Kernel Regression

In real word applications, data generating process for training a machin...
∙ 02/07/2020

Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks

A fundamental question in modern machine learning is how deep neural net...
∙ 02/02/2013

Generalization Guarantees for a Binary Classification Framework for Two-Stage Multiple Kernel Learning

We present generalization bounds for the TS-MKL framework for two stage ...
∙ 05/13/2022

Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime

The generalization performance of kernel ridge regression (KRR) exhibits...
∙ 10/26/2017

Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior

We describe an approach to understand the peculiar and counterintuitive ...
∙ 05/30/2022

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression

As modern machine learning models continue to advance the computational ...
∙ 06/28/2022

Target alignment in truncated kernel ridge regression

Kernel ridge regression (KRR) has recently attracted renewed interest du...

Code Repositories