A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel

06/25/2022
by   Mohamad Amin Mohamadi, et al.
0

Empirical neural tangent kernels (eNTKs) can provide a good understanding of a given network's representation: they are often far less expensive to compute and applicable more broadly than infinite width NTKs. For networks with O output units (e.g. an O-class classifier), however, the eNTK on N inputs is of size NO × NO, taking O((NO)^2) memory and up to O((NO)^3) computation. Most existing applications have therefore used one of a handful of approximations yielding N × N kernel matrices, saving orders of magnitude of computation, but with limited to no justification. We prove that one such approximation, which we call "sum of logits", converges to the true eNTK at initialization for any network with a wide final "readout" layer. Our experiments demonstrate the quality of this approximation for various uses across a range of settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/27/2022

On the Neural Tangent Kernel Analysis of Randomly Pruned Wide Neural Networks

We study the behavior of ultra-wide neural networks when their weights a...
research
03/09/2023

Kernel Regression with Infinite-Width Neural Networks on Millions of Examples

Neural kernels have drastically increased performance on diverse and non...
research
06/18/2020

The Recurrent Neural Tangent Kernel

The study of deep networks (DNs) in the infinite-width limit, via the so...
research
05/19/2022

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

We analyze feature learning in infinite width neural networks trained wi...
research
07/14/2022

Computing Optimal Kernels in Two Dimensions

Let P be a set of n points in ℝ^2. A subset C⊆ P is an ε-kernel of P if ...
research
02/14/2018

Optimal Transport: Fast Probabilistic Approximation with Exact Solvers

We propose a simple subsampling scheme for fast randomized approximate c...
research
12/19/2019

Integrated Emulators for Systems of Computer Models

We generalize the state-of-the-art linked emulator for a system of two c...

Please sign up or login with your details

Forgot password? Click here to reset