Generalized Leverage Score Sampling for Neural Networks

09/21/2020
by   Jason D. Lee, et al.
0

Leverage score sampling is a powerful technique that originates from theoretical computer science, which can be used to speed up a large number of fundamental questions, e.g. linear regression, linear programming, semi-definite programming, cutting plane method, graph sparsification, maximum matching and max-flow. Recently, it has been shown that leverage score sampling helps to accelerate kernel methods [Avron, Kapralov, Musco, Musco, Velingker and Zandieh 17]. In this work, we generalize the results in [Avron, Kapralov, Musco, Musco, Velingker and Zandieh 17] to a broader class of kernels. We further bring the leverage score sampling into the field of deep learning theory. ∙ We show the connection between the initialization for neural network training and approximating the neural tangent kernel with random features. ∙ We prove the equivalence between regularized neural network and neural tangent kernel ridge regression under the initialization of both classical random Gaussian and leverage score sampling.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/31/2018

On Fast Leverage Score Sampling and Optimal Learning

Leverage score sampling provides an appealing way to perform approximate...
research
05/21/2018

Relating Leverage Scores and Density using Regularized Christoffel Functions

Statistical leverage scores emerged as a fundamental tool for matrix ske...
research
06/15/2021

Scaling Neural Tangent Kernels via Sketching and Random Features

The Neural Tangent Kernel (NTK) characterizes the behavior of infinitely...
research
02/19/2018

Tail bounds for volume sampled linear regression

The n × d design matrix in a linear regression problem is given, but the...
research
08/06/2023

Gradient Coding through Iterative Block Leverage Score Sampling

We generalize the leverage score sampling sketch for ℓ_2-subspace embedd...
research
04/03/2021

Random Features for the Neural Tangent Kernel

The Neural Tangent Kernel (NTK) has discovered connections between deep ...
research
04/27/2019

Linearized two-layers neural networks in high dimension

We consider the problem of learning an unknown function f_ on the d-dime...

Please sign up or login with your details

Forgot password? Click here to reset