On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

by   Amnon Geifman, et al.

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.


page 1

page 2

page 3

page 4


A Kernel Perspective of Skip Connections in Convolutional Networks

Over-parameterized residual networks (ResNets) are amongst the most succ...

Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks

Deep residual network architectures have been shown to achieve superior ...

A Gaussian Process perspective on Convolutional Neural Networks

In this paper we cast the well-known convolutional neural network in a G...

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

Neural tangent kernel (NTK) is a powerful tool to analyze training dynam...

Propriety of the reference posterior distribution in Gaussian Process regression

In a seminal article, Berger, De Oliveira and Sansó (2001) compare sever...

Generalized Dilation Neural Networks

Vanilla convolutional neural networks are known to provide superior perf...

Eigenstripping, Spectral Decay, and Edge-Expansion on Posets

Fast mixing of random walks on hypergraphs has led to myriad breakthroug...

Please sign up or login with your details

Forgot password? Click here to reset