DeepAI
Log In Sign Up

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

03/17/2022
by   Amnon Geifman, et al.
13

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/27/2022

A Kernel Perspective of Skip Connections in Convolutional Networks

Over-parameterized residual networks (ResNets) are amongst the most succ...
04/07/2021

Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks

Deep residual network architectures have been shown to achieve superior ...
10/25/2018

A Gaussian Process perspective on Convolutional Neural Networks

In this paper we cast the well-known convolutional neural network in a G...
09/16/2022

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

Neural tangent kernel (NTK) is a powerful tool to analyze training dynam...
05/23/2018

Propriety of the reference posterior distribution in Gaussian Process regression

In a seminal article, Berger, De Oliveira and Sansó (2001) compare sever...
05/08/2019

Generalized Dilation Neural Networks

Vanilla convolutional neural networks are known to provide superior perf...
05/02/2022

Eigenstripping, Spectral Decay, and Edge-Expansion on Posets

Fast mixing of random walks on hypergraphs has led to myriad breakthroug...