Rate of Convergence of Polynomial Networks to Gaussian Processes

11/04/2021
by   Adam Klukowski, et al.
0

We examine one-hidden-layer neural networks with random weights. It is well-known that in the limit of infinitely many neurons they simplify to Gaussian processes. For networks with a polynomial activation, we demonstrate that the rate of this convergence in 2-Wasserstein metric is O(n^-1/2), where n is the number of hidden neurons. We suspect this rate is asymptotically sharp. We improve the known convergence rate for other activations, to power-law in n for ReLU and inverse-square-root up to logarithmic factors for erf. We explore the interplay between spherical harmonics, Stein kernels and optimal transport in the non-isotropic setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/17/2021

Non-asymptotic approximations of neural networks by Gaussian processes

We study the extent to which wide neural networks may be approximated by...
research
07/04/2021

Random Neural Networks in the Infinite Width Limit as Gaussian Processes

This article gives a new proof that fully connected neural networks with...
research
05/17/2023

Deep quantum neural networks form Gaussian processes

It is well known that artificial neural networks initialized from indepe...
research
11/18/2021

Bounds in L^1 Wasserstein distance on the normal approximation of general M-estimators

We derive quantitative bounds on the rate of convergence in L^1 Wasserst...
research
02/06/2020

Global Convergence of Frank Wolfe on One Hidden Layer Networks

We derive global convergence bounds for the Frank Wolfe algorithm when t...
research
08/03/2023

Memory capacity of two layer neural networks with smooth activations

Determining the memory capacity of two-layer neural networks with m hidd...
research
11/10/2022

On power sum kernels on symmetric groups

In this note, we introduce a family of "power sum" kernels and the corre...

Please sign up or login with your details

Forgot password? Click here to reset