Convergence and Sample Complexity of SGD in GANs

12/01/2020
by   Vasilis Kontonis, et al.
0

We provide theoretical convergence guarantees on training Generative Adversarial Networks (GANs) via SGD. We consider learning a target distribution modeled by a 1-layer Generator network with a non-linear activation function ϕ(·) parametrized by a d × d weight matrix 𝐖_*, i.e., f_*(𝐱) = ϕ(𝐖_* 𝐱). Our main result is that by training the Generator together with a Discriminator according to the Stochastic Gradient Descent-Ascent iteration proposed by Goodfellow et al. yields a Generator distribution that approaches the target distribution of f_*. Specifically, we can learn the target distribution within total-variation distance ϵ using Õ(d^2/ϵ^2) samples which is (near-)information theoretically optimal. Our results apply to a broad class of non-linear activation functions ϕ, including ReLUs and is enabled by a connection with truncated statistics and an appropriate design of the Discriminator network. Our approach relies on a bilevel optimization framework to show that vanilla SGDA works.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/09/2020

When can Wasserstein GANs minimize Wasserstein Distance?

Generative Adversarial Networks (GANs) are widely used models to learn c...
research
10/15/2019

SGD Learns One-Layer Networks in WGANs

Generative adversarial networks (GANs) are a widely used framework for l...
research
02/10/2020

Statistical Guarantees of Generative Adversarial Networks for Distribution Estimation

Generative Adversarial Networks (GANs) have achieved great success in un...
research
06/09/2019

The Implicit Metropolis-Hastings Algorithm

Recent works propose using the discriminator of a GAN to filter out unre...
research
07/14/2017

f-GANs in an Information Geometric Nutshell

Nowozin et al showed last year how to extend the GAN principle to all f-...
research
01/27/2019

Deconstructing Generative Adversarial Networks

We deconstruct the performance of GANs into three components: 1. Formu...
research
09/27/2019

On the Anomalous Generalization of GANs

Generative models, especially Generative Adversarial Networks (GANs), ha...

Please sign up or login with your details

Forgot password? Click here to reset