Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators

12/27/2021
by   Carles Domingo Enrich, et al.
0

We construct pairs of distributions μ_d, ν_d on ℝ^d such that the quantity |𝔼_x ∼μ_d [F(x)] - 𝔼_x ∼ν_d [F(x)]| decreases as Ω(1/d^2) for some three-layer ReLU network F with polynomial width and weights, while declining exponentially in d if F is any two-layer network with polynomial weights. This shows that deep GAN discriminators are able to distinguish distributions that shallow discriminators cannot. Analogously, we build pairs of distributions μ_d, ν_d on ℝ^d such that |𝔼_x ∼μ_d [F(x)] - 𝔼_x ∼ν_d [F(x)]| decreases as Ω(1/(dlog d)) for two-layer ReLU networks with polynomial weights, while declining exponentially for bounded-norm functions in the associated RKHS. This confirms that feature learning is beneficial for discriminators. Our bounds are based on Fourier transforms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/20/2023

Any Deep ReLU Network is Shallow

We constructively prove that every deep ReLU network can be rewritten as...
research
10/07/2021

Tighter Sparse Approximation Bounds for ReLU Neural Networks

A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski Ba...
research
10/10/2018

Random ReLU Features: Universality, Approximation, and Composition

We propose random ReLU features models in this work. Its motivation is r...
research
06/19/2019

Disentangling feature and lazy learning in deep neural networks: an empirical study

Two distinct limits for deep learning as the net width h→∞ have been pro...
research
02/13/2019

How do infinite width bounded norm networks look in function space?

We consider the question of what functions can be captured by ReLU netwo...
research
10/02/2019

Identifying Weights and Architectures of Unknown ReLU Networks

The output of a neural network depends on its parameters in a highly non...
research
06/10/2021

Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

Several works in implicit and explicit generative modeling empirically o...

Please sign up or login with your details

Forgot password? Click here to reset