DeepAI AI Chat
Log In Sign Up

Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

by   Arda Sahiner, et al.

Generative Adversarial Networks (GANs) are commonly used for modeling complex distributions of data. Both the generators and discriminators of GANs are often modeled by neural networks, posing a non-transparent optimization problem which is non-convex and non-concave over the generator and discriminator, respectively. Such networks are often heuristically optimized with gradient descent-ascent (GDA), but it is unclear whether the optimization problem contains any saddle points, or whether heuristic methods can find them in practice. In this work, we analyze the training of Wasserstein GANs with two-layer neural network discriminators through the lens of convex duality, and for a variety of generators expose the conditions under which Wasserstein GANs can be solved exactly with convex optimization approaches, or can be represented as convex-concave games. Using this convex duality interpretation, we further demonstrate the impact of different activation functions of the discriminator. Our observations are verified with numerical results demonstrating the power of the convex interpretation, with applications in progressive training of convex architectures corresponding to linear generators and quadratic-activation discriminators for CelebA image generation. The code for our experiments is available at


page 9

page 16

page 17


Understanding Overparameterization in Generative Adversarial Networks

A broad class of unsupervised deep learning methods such as Generative A...

SGD Learns One-Layer Networks in WGANs

Generative adversarial networks (GANs) are a widely used framework for l...

Evaluating GANs via Duality

Generative Adversarial Networks (GANs) have shown great results in accur...

Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex

Several recently proposed architectures of neural networks such as ResNe...

Data Interpolants – That's What Discriminators in Higher-order Gradient-regularized GANs Are

We consider the problem of optimizing the discriminator in generative ad...

The Curious Case of Convex Networks

In this paper, we investigate a constrained formulation of neural networ...

Generative Adversarial Method Based on Neural Tangent Kernels

The recent development of Generative adversarial networks (GANs) has dri...