Implementation BEGAN([Boundary Equilibrium Generative Adversarial Networks](https://arxiv.org/pdf/1703.10717.pdf)) by Keras.
We propose a new equilibrium enforcing method paired with a loss derived from the Wasserstein distance for training auto-encoder based Generative Adversarial Networks. This method balances the generator and discriminator during training. Additionally, it provides a new approximate convergence measure, fast and stable training and high visual quality. We also derive a way of controlling the trade-off between image diversity and visual quality. We focus on the image generation task, setting a new milestone in visual quality, even at higher resolutions. This is achieved while using a relatively simple model architecture and a standard training procedure.READ FULL TEXT VIEW PDF
Implementation BEGAN([Boundary Equilibrium Generative Adversarial Networks](https://arxiv.org/pdf/1703.10717.pdf)) by Keras.
My re-implementation of the BEGAN by Berthelot et al. with some modifications.
Surya Dantuluri's HAILCon Presentation on GANs alongside Jeff Dean (Google Senior Fellow), Peter Norvig (Director of Research @ Google), and Stefano Ermon (Professor @ Stanford University)
This is an implementation of https://arxiv.org/pdf/1703.10717.pdf
Boundary Equilibrium GAN
Generative Adversarial Networks (GANs) are a class of methods for learning a data distribution and realizing a model to sample from it. GANs are architectured around two functions: the generator , which maps a sample
from a random uniform distribution to the data distribution, and the discriminatorwhich determines if a sample belongs to the data distribution. The generator and discriminator are typically learned jointly by alternating the training of and
, based on game theory principles.
GANs can generate very convincing images, sharper than ones produced by auto-encoders using pixel-wise losses. However, GANs still face many unsolved difficulties: in general they are notoriously difficult to train, even with many tricks applied [15, 16]. Correct hyper-parameter selection is critical. Controlling the image diversity of the generated samples is difficult. Balancing the convergence of the discriminator and of the generator is a challenge: frequently the discriminator wins too easily at the beginning of training . GANs easily suffer from modal collapse, a failure mode in which just one image is learned 
. Heuristic regularizers such as batch discrimination and the repelling regularizer  have been proposed to alleviate this problem with varying degrees of success.
In this paper, we make the following contributions:
A GAN with a simple yet robust architecture, standard training procedure with fast and stable convergence.
An equilibrium concept that balances the power of the discriminator against the generator.
A new way to control the trade-off between image diversity and visual quality.
An approximate measure of convergence. To our knowledge the only other published measure is from Wasserstein GAN  (WGAN), which will be discussed in the next section.
Deep Convolutional GANs (DCGANs) first introduced a convolutional architecture which led to improved visual quality. More recently, Energy Based GANs (EBGANs) were proposed as a class of GANs that aims to model the discriminator as an energy function. This variant converges more stably and is both easy to train and robust to hyper-parameter variations. The authors attribute some of these benefits to the larger number of targets in the discriminator. EBGAN likewise implements its discriminator as an auto-encoder with a per-pixel error.
While earlier GAN variants lacked a measure of convergence, Wasserstein GANs  (WGANs) recently introduced a loss that also acts as a measure of convergence. In their implementation it comes at the expense of slow training, but with the benefit of stability and better mode coverage.
We use an auto-encoder as a discriminator as was first proposed in EBGAN 
. While typical GANs try to match data distributions directly, our method aims to match auto-encoder loss distributions using a loss derived from the Wasserstein distance. This is done using a typical GAN objective with the addition of an equilibrium term to balance the discriminator and the generator. Our method has an easier training procedure and uses a simpler neural network architecture compared to typical GAN techniques.
We wish to study the effect of matching the distribution of the errors instead of matching the distribution of the samples directly. We first introduce the auto-encoder loss, then we compute a lower bound to the Wasserstein distance between the auto-encoder loss distributions of real and generated samples.
We first introduce
the loss for training a pixel-wise autoencoder as:
Let be two distributions of auto-encoder losses, let be the set all of couplings of and , and let be their respective means. The Wasserstein distance can be expressed as:
Using Jensen’s inequality, we can derive a lower bound to :
It is important to note that we are aiming to optimize a lower bound of the Wasserstein distance between auto-encoder loss distributions, not between sample distributions.
We design the discriminator to maximize equation 1 between auto-encoder losses. Let be the distribution of the loss , where are real samples. Let be the distribution of the loss , where is the generator function and are uniform random samples of dimension .
Since there are only two possible solutions to maximizing :
We select solution (b) for our objective since minimizing leads naturally to auto-encoding the real images. Given the discriminator and generator parameters and , each updated by minimizing the losses and , we express the problem as the GAN objective, where and are samples from :
Note that in the following we use an abbreviated notation: and .
This equation, while similar to the one from WGAN , has two important differences: First we match distributions between losses, not between samples. And second, we do not explicitly require the discriminator to be K-Lipschitz since we are not using the Kantorovich and Rubinstein duality theorem .
For function approximations, in our case deep neural networks, we must also consider the representational capacities of each function and . This is determined both by the model implementing the function and the number of parameters. It is typically the case that and are not well balanced and the discriminator wins easily. To account for this situation we introduce an equilibrium concept.
In practice it is crucial to maintain a balance between the generator and discriminator losses; we consider them to be at equilibrium when:
If we generate samples that cannot be distinguished by the discriminator from real ones, the distribution of their errors should be the same, including their expected error. This concept allows us to balance the effort allocated to the generator and discriminator so that neither wins over the other.
We can relax the equilibrium with the introduction of a new hyper-parameter defined as
In our model, the discriminator has two competing goals: auto-encode real images and discriminate real from generated images. The term lets us balance these two goals. Lower values of lead to lower image diversity because the discriminator focuses more heavily on auto-encoding real images. We will refer to as the diversity ratio. There is a natural boundary for which images are sharp and have details.
The BEGAN objective is:
We use Proportional Control Theory to maintain the equilibrium . This is implemented using a variable to control how much emphasis is put on during gradient descent. We initialize . is the proportional gain for
; in machine learning terms, it is the learning rate for. We used 0.001 in our experiments. In essence, this can be thought of as a form of closed-loop feedback control in which is adjusted at each step to maintain equation 4.
In early training stages, tends to generate easy-to-reconstruct data for the auto-encoder since generated data is close to 0 and the real data distribution has not been learned accurately yet. This yields to early on and this is maintained for the whole training process by the equilibrium constraint.
The introductions of the approximation in equation 1 and in equation 4 have an impact on our modeling of the Wasserstein distance. Consequently, examination of samples generated from various values is of primary interest as will be shown in the results section.
In contrast to traditional GANs which require alternating training and , or pretraining , our proposed method BEGAN requires neither to train stably. Adam  was used during training with the default hyper-parameters. and are updated independently based on their respective losses with separate Adam optimizers. We typically used a batch size of .
Determining the convergence of GANs is generally a difficult task since the original formulation is defined as a zero-sum game. As a consequence, one loss goes up when the other goes down. The number of epochs or visual inspection are typically the only practical ways to get a sense of how training has progressed.
We derive a global measure of convergence by using the equilibrium concept: we can frame the convergence process as finding the closest reconstruction with the lowest absolute value of the instantaneous process error for the proportion control algorithm . This measure is formulated as the sum of these two terms:
This measure can be used to determine when the network has reached its final state or if the model has collapsed.
The discriminator is a convolutional deep neural network architectured as an auto-encoder. is shorthand for the dimensions of where are the height, width and colors. We use an auto-encoder with both a deep encoder and decoder. The intent is to be as simple as possible to avoid typical GAN tricks.
(ELUs) applied at their outputs. Each layer is repeated a number of times (typically 2). We observed that more repetitions led to even better visual results. The convolution filters are increased linearly with each down-sampling. Down-sampling is implemented as sub-sampling with stride 2 and up-sampling is done by nearest neighbor. At the boundary between the encoder and the decoder, the tensor of processed data is mapped via fully connected layers, not followed by any non-linearities, to and from an embedding statewhere is the dimension of the auto-encoder’s hidden state.
The generator uses the same architecture (though not the same weights) as the discriminator decoder. We made this choice only for simplicity. The input state is sampled uniformly.
This simple architecture achieves high quality results and demonstrates the robustness of our technique.
Further, optional, refinements aid gradient propagation and produce yet sharper images. Taking inspiration from deep residual networks , we initialize the network using vanishing residuals: for successive same sized layers, the layer’s input is combined with its output: . In our experiments, we start with and progressively decrease it to 0 over 16000 steps (one epoch).
We also introduce skip connections [8, 17, 9] to help gradient propagation . The first decoder tensor is obtained from projecting to an tensor. After each upsampling step, the output is concatenated with upsampled to the same dimensions. This creates a skip connection between the hidden state and each successive upsampling layer of the decoder.
We did not explore other techniques typically used in GANs, such as batch normalization, dropout, transpose convolutions or exponential growth for convolution filters, though they might further improve upon these results.
We trained our model using Adam with an initial learning rate of , decaying by a factor of when the measure of convergence stalls. Modal collapses or visual artifacts were observed sporadically with high initial learning rates, however simply reducing the learning rate was sufficient to avoid them. We trained models for varied resolutions from to , adding or removing convolution layers to adjust for the image size, keeping a constant final down-sampled image size of . We used in most of our experiments with this dataset.
Our biggest model for images used a convolution with filters and had a total of trainable parameters. Training time was about days on four P100 GPUs. Smaller models of size could train in a few hours on a single GPU.
We use a dataset of celebrity face images for training in place of CelebA 
. This dataset has a larger variety of facial poses, including rotations around the camera axis. These are more varied and potentially more difficult to model than the aligned faces from CelebA, presenting an interesting challenge. We preferred the use of faces as a visual estimator since humans excel at identifying flaws in faces.
Figure (b)b shows some representative samples drawn uniformly from at resolutions of . Higher resolution images, while maintaining coherency, tend to lose sharpness, but this may be improved upon with additional hyper-parameter explorations. To our knowledge these are the first anatomically coherent high-resolution results except for Stacked GANs  which has shown some promise for flowers and birds at up to .
We observe varied poses, expressions, genders, skin colors, light exposure, and facial hair. However we did not see glasses, we see few older people and there are more women than men. For comparison we also displayed some EBGAN  results in figure (a)a. We must keep in mind that these are trained on different datasets so direct comparison is difficult.
In Figure 3, we compared the effect of varying . The model appears well behaved, still maintaining a degree of image diversity across the range of values. At low values, the faces look overly uniform. Variety increases with but so do artifacts. Our observations seem to contradict those of  that diversity and quality were independent.
To estimate the modal coverage of our generator we take real images and find their corresponding embedding for the generator. This is done using Adam to find a value for that minimizes
. Mapping to real images is not the goal of the model but it provides a way of testing its ability to generalize. By interpolating theembeddings between two real images, we verify that the model generalized the image contents rather than simply memorizing them.
Figure (c)c displays interpolations on between real images at resolution; these images were not part of the training data. The first and last columns contain the real images to be represented and interpolated. The images immediately next to them are their corresponding approximations while the images in-between are the results of linear interpolation in . For comparison with the current state of the art for generative models, we included ALI  results at (figure (a)a) and conditional PixelCNN  results at (figure (b)b) both trained on different data sets (higher resolutions were not available to us for these models). In addition figure (d)d showcases interpolation between an image and its mirror.
Sample diversity, while not perfect, is convincing; the generated images look relatively close to the real ones. The interpolations show good continuity. On the first row, the hair transitions in a natural way and intermediate hairstyles are believable, showing good generalization. It is also worth noting that some features are not represented such as the cigarette in the left image. The second and last rows show simple rotations. While the rotations are smooth, we can see that profile pictures are not captured as well as camera facing ones. We assume this is due to profiles being less common in our dataset. Finally the mirror example demonstrates separation between identity and rotation. A surprisingly realistic camera-facing image is derived from a single profile image.
The convergence measure was conjectured earlier to measure the convergence of the BEGAN model. As can be seen in figure 5 this measure correlates well with image fidelity. We can also see from this plot that the model converges quickly, just as was originally reported for EBGANs. This seems to confirm the fast convergence property comes from pixel-wise losses.
To test the robustness of the equilibrium balancing technique, we performed an experiment advantaging the discriminator over the generator, and vice versa. Figure 6 displays the results.
By maintaining the equilibrium the model remained stable and converged to meaningful results. The image quality suffered as expected with low dimensionality of due to the reduced capacity of the discriminator. Surprisingly, reducing the dimensionality of had relatively little effect on image diversity or quality.
|Improved GANs ||4.||36|
|MIX + WGAN ||4.||04|
To measure quality and diversity numerically, we computed the inception score  on CIFAR-10 images. The inception score is a heuristic that has been used for GANs to measure single sample quality and diversity on the inception model. We train an unconditional version of our model and compare to previous unsupervised results. The goal is to generate a distribution that is representative of the original data.
A comparison to similar works on models trained entirely unsupervised is shown in table 1. With the exception of Denoising Feature Matching  (DFM), our score is better than other GAN techniques that directly aim to match the data distribution. This seems to confirm experimentally that matching loss distributions of the auto-encoder is an effective indirect method of matching data distributions. DFM appears compatible with our method and combining them is a possible avenue for future work.
There are still many unexplored avenues. Does the discriminator have to be an auto-encoder? Having pixel-level feedback seems to greatly help convergence, however using an auto-encoder has its drawbacks: what latent space size is best for a dataset? When should noise be added to the input and how much? What impact would using other varieties of auto-encoders such Variational Auto-Encoders (VAEs) have?
More fundamentally, we note that our objective bears a superficial resemblance to the WGAN  objective. Is the auto-encoder combined with the equilibrium concept fulfilling a similar bounding functionality as the K-Lipschitz constraint in the WGAN formulation?
We introduced BEGAN, a GAN that uses an auto-encoder as the discriminator. Using proportional control theory, we proposed a novel equilibrium method for balancing adversarial networks. We believe this method has many potential applications such as dynamically weighing regularization terms or other heterogeneous objectives. Using this equilibrium method, the network converges to diverse and visually pleasing images. This remains true at higher resolutions with trivial modifications. Training is stable, fast and robust to parameter changes. It does not require a complex alternating training procedure. Our approach provides at least partial solutions to some outstanding GAN problems such as measuring convergence, controlling distributional diversity and maintaining the equilibrium between the discriminator and the generator. While we could partially control the diversity of generator by influencing the discriminator, there is clearly still room for improvement.
We would like to thank Jay Han, Llion Jones and Ankur Parikh for their help with the manuscript, Jakob Uszkoreit for his constant support, Wenze Hu, Aaron Sarna and Florian Schroff for technical support. Special thanks to Grant Reaber for his in-depth feedback on Wasserstein distance computation.