Reducing Noise in GAN Training with Variance Reduced Extragradient

by   Tatjana Chavdarova, et al.

Using large mini-batches when training generative adversarial networks (GANs) has been recently shown to significantly improve the quality of the generated samples. This can be seen as a simple but computationally expensive way of reducing the noise of the gradient estimates. In this paper, we investigate the effect of the noise in this context and show that it can prevent the convergence of standard stochastic game optimization methods, while their respective batch version converges. To address this issue, we propose a variance-reduced version of the stochastic extragradient algorithm (SVRE). We show experimentally that it performs similarly to a batch method, while being computationally cheaper, and show its theoretical convergence, improving upon the best rates proposed in the literature. Experiments on several datasets show that SVRE improves over baselines. Notably, SVRE is the first optimization method for GANs to our knowledge that can produce near state-of-the-art results without using adaptive step-size such as Adam.



There are no comments yet.


page 1

page 2

page 3

page 4


A Variational Inequality Perspective on Generative Adversarial Nets

Stability has been a recurrent issue in training generative adversarial ...

Taming GANs with Lookahead

Generative Adversarial Networks are notoriously challenging to train. Th...

A Variance Controlled Stochastic Method with Biased Estimation for Faster Non-convex Optimization

In this paper, we proposed a new technique, variance controlled stochast...

Kernel-Based Training of Generative Networks

Generative adversarial networks (GANs) are designed with the help of min...

Improved BiGAN training with marginal likelihood equalization

We propose a novel training procedure for improving the performance of g...

An Investigation into the Stochasticity of Batch Whitening

Batch Normalization (BN) is extensively employed in various network arch...

Small-GAN: Speeding Up GAN Training Using Core-sets

Recent work by Brock et al. (2018) suggests that Generative Adversarial ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.