Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity

10/22/2020
by   Shuxiao Chen, et al.
0

As a popular approach to modeling the dynamics of training overparametrized neural networks (NNs), the neural tangent kernels (NTK) are known to fall behind real-world NNs in generalization ability. This performance gap is in part due to the label agnostic nature of the NTK, which renders the resulting kernel not as locally elastic as NNs <cit.>. In this paper, we introduce a novel approach from the perspective of label-awareness to reduce this gap for the NTK. Specifically, we propose two label-aware kernels that are each a superimposition of a label-agnostic part and a hierarchy of label-aware parts with increasing complexity of label dependence, using the Hoeffding decomposition. Through both theoretical and empirical evidence, we show that the models trained with the proposed kernels better simulate NNs in terms of generalization ability and local elasticity.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/14/2022

Model Generalization: A Sharpness Aware Optimization Perspective

Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimiza...
research
09/11/2019

Automated Spectral Kernel Learning

The generalization performance of kernel methods is largely determined b...
research
10/27/2020

Toward Better Generalization Bounds with Locally Elastic Stability

Classical approaches in learning theory are often seen to yield very loo...
research
10/23/2020

An Investigation of how Label Smoothing Affects Generalization

It has been hypothesized that label smoothing can reduce overfitting and...
research
03/07/2022

Generalization Through The Lens Of Leave-One-Out Error

Despite the tremendous empirical success of deep learning models to solv...
research
02/12/2023

Generalization Ability of Wide Neural Networks on ℝ

We perform a study on the generalization ability of the wide two-layer R...
research
09/30/2020

Improving Generalization of Deep Fault Detection Models in the Presence of Mislabeled Data

Mislabeled samples are ubiquitous in real-world datasets as rule-based o...

Please sign up or login with your details

Forgot password? Click here to reset