DeepAI AI Chat
Log In Sign Up

On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures

by   Maxim Samarin, et al.

The Neural Tangent Kernel (NTK) is an important milestone in the ongoing effort to build a theory for deep learning. Its prediction that sufficiently wide neural networks behave as kernel methods, or equivalently as random feature models, has been confirmed empirically for certain wide architectures. It remains an open question how well NTK theory models standard neural network architectures of widths common in practice, trained on complex datasets such as ImageNet. We study this question empirically for two well-known convolutional neural network architectures, namely AlexNet and LeNet, and find that their behavior deviates significantly from their finite-width NTK counterparts. For wider versions of these networks, where the number of channels and widths of fully-connected layers are increased, the deviation decreases.


Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?

Neural Tangent Kernel (NTK) theory is widely used to study the dynamics ...

Asymptotics of Wide Convolutional Neural Networks

Wide neural networks have proven to be a rich class of architectures for...

Neural networks for classification of strokes in electrical impedance tomography on a 3D head model

We consider the problem of the detection of brain hemorrhages from three...

Kernel-based Translations of Convolutional Networks

Convolutional Neural Networks, as most artificial neural networks, are c...

On Exact Computation with an Infinitely Wide Neural Net

How well does a classic deep net architecture like AlexNet or VGG19 clas...

Deep Networks and the Multiple Manifold Problem

We study the multiple manifold problem, a binary classification task mod...

Building effective deep neural network architectures one feature at a time

Successful training of convolutional neural networks is often associated...