Taming GANs with Lookahead

06/25/2020
by   Tatjana Chavdarova, et al.
12

Generative Adversarial Networks are notoriously challenging to train. The underlying minimax optimization is highly susceptible to the variance of the stochastic gradient and the rotational component of the associated game vector field. We empirically demonstrate the effectiveness of the Lookahead meta-optimization method for optimizing games, originally proposed for standard minimization. The backtracking step of Lookahead naturally handles the rotational game dynamics, which in turn enables the gradient ascent descent method to converge on challenging toy games often analyzed in the literature. Moreover, it implicitly handles high variance without using large mini-batches, known to be essential for reaching state of the art performance. Experimental results on MNIST, SVHN, and CIFAR-10, demonstrate a clear advantage of combining Lookahead with Adam or extragradient, in terms of performance, memory footprint, and improved stability. Using 30-fold fewer parameters and 16-fold smaller minibatches we outperform the reported performance of the class-dependent BigGAN on CIFAR-10 by obtaining FID of 13.65without using the class labels, bringing state-of-the-art GAN training within reach of common computational resources.

READ FULL TEXT

page 24

page 25

page 26

page 27

page 28

page 29

research
10/29/2022

Recursive Reasoning in Minimax Games: A Level k Gradient Play Method

Despite the success of generative adversarial networks (GANs) in generat...
research
04/18/2019

Reducing Noise in GAN Training with Variance Reduced Extragradient

Using large mini-batches when training generative adversarial networks (...
research
07/12/2018

Negative Momentum for Improved Game Dynamics

Games generalize the optimization paradigm by introducing different obje...
research
05/23/2022

HessianFR: An Efficient Hessian-based Follow-the-Ridge Algorithm for Minimax Optimization

Wide applications of differentiable two-player sequential games (e.g., i...
research
06/12/2018

The Unusual Effectiveness of Averaging in GAN Training

We show empirically that the optimal strategy of parameter averaging in ...
research
11/10/2021

Training Generative Adversarial Networks with Adaptive Composite Gradient

The wide applications of Generative adversarial networks benefit from th...

Please sign up or login with your details

Forgot password? Click here to reset