1 Introduction
Deep generative models have gained widespread use as a tractable way to model highdimensional data distributions. Recently introduced frameworks, such as generative adversarial networks (GANs) goodfellow2014
and variational autoencoders (VAEs)
kingma2013, can map a noise distribution to the data space, producing realistic samples. Capturing these highdimensional data distributions makes possible a variety of downstream tasks, e.g., semisupervised learning, sampling, and reconstruction/denoising.
These models are also amenable to conditional training, where a conditional input (e.g., class label, data from another modality) guides the sampling process mirza2014 ; sohn2015 . At test time, manually selected conditional inputs lead to samples from the corresponding conditional distribution. Assuming the distributions share some underlying structure, this vastly reduces the capacity required to model the distributions compared to modeling each conditional distribution in a separate network.
However, the standard training regime for deep generative models assumes that training data is drawn i.i.d. from the distribution of interest. For example, in the setting where the dataset contains multiple classes (e.g., MNIST digits), data representing every class is used concurrently during training. In many realworld scenarios, data may actually arrive sequentially or only be available for a short time period.
Notably, a number of potential applications of deep generative models encounter exactly this constraint. One reason for widespread interest in unsupervised learning is that vastly more unlabeled data is available than labeled data. In many cases, such as an agent exploring an environment and learning a visual representation via a camera feed or training a generative model of text from the Twitter firehose, it is unlikely to be practical to save all the data over the lifetime of the system. A (batched) online or streaming approach is preferable.
Another area which has gained recent interest and which exhibits this constraint is private learning dldp
. Learning a differentially private deep classifier from many private datasets currently requires many rounds of communication with or queries to models trained locally on each dataset
papernot2016semi ; federated . While private learning of deep generative models is still an open research problem, a differentially private deep generative model would allow generation of synthetic data and good feature representations for downstream tasks, without the need for repeated communication. Given this model, it would be desirable to efficiently update it from private data sources in sequence without having to access previous data sources again.If we wish to update a trained generative model to capture a newly observed distribution, naively training the model solely on the new data will result in the previously learned distributions being forgotten (Figure 1
). This is due to the general susceptibility of neural networks to catastrophic forgetting when trained on multiple tasks sequentially
mccloskey1989 . Instead, the standard training regime requires that we train on new and old data simultaneously. This approach is not very scalable as it requires that all previously observed data be stored, or synthetic data representative of previous observations be regenerated, for each round of training.In this work, we propose a scalable approach to training generative adversarial nets, a prominent deep generative model, on multiple datasets sequentially, where training data for any previous dataset is assumed to be inaccessible. We leverage recent work with discriminative models where synaptic plasticity is reduced for network parameters determined to be critical for previously encountered tasks kirkpatrick2016 ; zenke2017 . Experimental evaluation of our approach demonstrates that GANs may be extended to settings where the observed data distribution changes over time, enabling continual learning of novel generative tasks.
2 Deep generative models
In generative modeling, we wish to learn some distribution , e.g., the distribution of natural images. In a deep generative model,
is approximated by a neural network. Some generative models may be structured to allow for explicit estimation of the likelihood by directly approximating
with a function . Such models include variational autoencoders kingma2013 and their variants. Other generative models only allow for simulation of the data for sampling via a trainable, onedirectional mapping from a fixed prior on latent variables to the generator’s output distribution:(1) 
In this case, the induced density is , where . Such models notably include generative adversarial nets goodfellow2014 . While in this work we present a method for continual learning with GANs, we believe that similar routes may be taken with other generative modeling frameworks.
2.1 Generative adversarial nets
The standard GAN framework pits a generator against a discriminator . Given a random noise input drawn from some prior
(typically a uniform or Gaussian distribution),
deterministically generates a sample (e.g., an image). The generator’s goal is to fool the discriminator such that the discriminator cannot determine if the generated samples come from , ’s current learned distribution, or , the true data distribution. and are trained according to the following twoplayer minimax objective from goodfellow2014 :(2) 
2.2 Conditional generative adversarial nets
In mirza2014 , the GAN framework is extended to allow both and to be conditioned on an extra input . The minimax objective then becomes:
(3) 
This conditional framework is quite flexible, both in terms of what the input can be and how it enters the generator and discriminator networks. For example,
may range from being a discrete, onehot vector representing a class label to being a partially complete sample, as in the case of image inpainting. Furthermore,
may simply be included as input to the first layers of and, or it may be input at every layer. This is considered an architectural hyperparameter, and certain ways of incorporating
may be more effective than others depending on the specific problem domain.3 Catastrophic forgetting and remedies
When a neural network is trained on multiple tasks sequentially, the process of training on the later tasks tends to decrease performance on the earlier tasks. This phenomenon, known as catastrophic forgetting, was first explored in mccloskey1989
. A classic experiment in that work consisted of first training a multilayer perceptron with backpropagation on "ones" addition tasks (e.g.,
,) until it learned to correctly compute the sums. Then, the network was trained on "twos" addition tasks, without further access to the training data or loss function for ones addition. As the network learned to perform twos addition, it forgot how to performs ones addition.
This same phenomenon is observable across diverse domains in which neural networks are widely used, including computer vision, speech recognition, and natural language processing. When network weights can adjust without restriction to meet the objective of the current task, previous task performance is severely disrupted.
To our knowledge, existing literature on preventing catastrophic forgetting in neural networks has focused on discriminative/classification settings. Some approaches aim to directly preserve the functional input/output mapping of the network for previous tasks. For example, in li2016 , when new task training data arrives, the softmax activations of previous task output layers are recorded for all new training examples. Then, in a similar process to knowledge distillation hinton2015 , these softmax activations serve as targets for the old output layers while training on the new task. Other approaches make dramatic architectural changes to reduce interference between different tasks. For example, in rusu2016 , distinct columns are added to a network for each task while freezing parameters in all previous columns. Lateral connections allow the sharing of features between tasks. While forgetting is prevented, the number of parameters grows quadratically in the number of tasks. In our work, we assume a setting where even linear growth in the number of parameters is undesirable.
3.1 Protecting critical parameters
Kirkpatrick et al. kirkpatrick2016 recently proposed an approach, dubbed elastic weight consolidation (EWC), that is computationally inexpensive and does not require adding any new capacity to the network. Those parameters found to be critical for the performance achieved on previous tasks are protected with an additional loss term restricting their movement. Given a network parameterized by trained on some task with training data , the posterior distribution of the parameters, , is approximated as a Gaussian distribution with mean
(the actual parameter values after training) and variance given by the inverse diagonal of the Fisher information matrix
(or Hessian). Thus those parameters associated with higher Fisher diagonal elements are considered to be more "certain" and thus less flexible. When the network begins training on a new task without further access to task ’s training data, the loss function for task is augmented as:(4) 
where is the standard loss for task , is the diagonal element of the Fisher information matrix, and is a hyperparameter denoting the relative importance of the tasks. Experimental results in kirkpatrick2016
on permuted MNIST tasks and Atari games validated this approach for classification tasks in supervised and reinforcement learning settings.
A similar method is proposed in zenke2017 . Rather than compute the weight saliency at the end of a task’s training session, an online measure takes into account a weight’s total contribution to the global change in loss over a training trajectory. In this case, those weights with higher contributions are more critical. When moving to subsequent tasks, a quadratic penalty is used similar to kirkpatrick2016 .
These approaches have connections to lecun1990 , where the second derivative of the objective function with respect to the network parameters, in combination with parameter magnitude, is used to estimate saliency. Low saliency parameters are pruned, leading to reduced network size. Rather than completely removing low saliency parameters, kirkpatrick2016 ; zenke2017 allow such parameters to have greater flexibility when learning a new task.
4 Continual learning with GANs
Assume we have a generator trained to simulate some distribution of interest. When the observed data distribution changes, we may wish to additionally model this new distribution without interfering with the old. While we could certainly train an entirely new GAN, for similar distributions (e.g., images of cats and images of dogs), this approach prevents the leveraging of any shared underlying structure. In addition, storing separate models for each task is not scalable in limited capacity settings.
One might try to simply continue training the model according to the standard objective, only using data from the new distribution. If the network trains according to this setup however, it quickly undergoes catastrophic forgetting (Figure 1). Without being exposed to real examples from the previously encountered data, learns to determine if its inputs are real examples sampled from the new distribution only. Therefore, all of ’s resources will be directed towards fooling solely on the new task, leading to an inability to sample from the previously learned distributions.
A simple yet expensive solution to ’s forgetting problem is to regenerate a training set for each of ’s previously encountered datasets, a route unavailable in the discriminate learning setting. Assuming has been trained well, the samples it generates should be realistic enough to serve as "real" inputs to . Then standard training can proceed simultaneously on all the data, both new and old.
While this route reduces the extent of catastrophic forgetting, it is not very efficient. For the observed distribution, we must regenerate training sets for the previous distributions. In addition, we must retrain on all of the regenerated data in addition to the newly observed data. Learning from a sequence of observed distributions would then have a time complexity of . Instead, the approach we ultimately take is , requiring no regeneration of training data.
4.1 Augmented generator objective
We propose to replace ’s portion of the standard minimax objective with an augmented loss function discouraging those weights critical for accurately modeling the previously observed distribution(s) from undergoing drastic changes in value. However, in the unconditional GAN framework, this constraint would attempt to preserve the mapping from the single noise prior to the previous data distribution, while ’s objective would encourage mapping the noise prior to the new data distribution. We observed that is able to overcome arbitrarily large fixed constraints, ultimately resulting in the forgetting of ’s previously learned distribution.
In the conditional GAN framework, the distinct observed data distributions may be associated with distinct conditional inputs, maintaining a welldefined desired mapping from the joint distribution of
and a conditional input to the data space. In settings where does not change over time but does, such as when datasets corresponding to each class are observed sequentially, we may directly apply the constraints to a conditional GAN. In settings where conditional inputs are not available or the observed distribution changes arbitrarily over time, we may artificially assign a distinct conditional input for the distributions observed at different times.The portion of the standard minimax objective for conditional GANs relevant to determining ’s parameter updates is:
(5) 
In practice, we follow the standard approach of using as the generator’s cost function instead to provide better gradients early in training.
When has been trained well on an initial set of conditional inputs leading to optimal parameters for task , we do not want subsequent task training sessions to significantly damage the performance achieved. Intuitively, the distribution of should not be greatly altered for task . Estimating the Fisher information of with respect to ’s parameters allows us to determine those that are most critical to fooling , and we can correspondingly penalize the movement of these parameters. Constraining movement of parameters with high Fisher information for previous tasks was introduced for discriminative models in kirkpatrick2016 as elastic weight consolidation (EWC).
We define the penalty weighting for ’s parameter as:
(6) 
Those parameters that, if perturbed, would have a high effect on the discriminator’s output have a high value for the corresponding element in the diagonal of the Fisher information matrix and are considered especially salient. Note that the above actually corresponds to an approximation known as the empirical Fisher information, which is more efficient to compute martens2016 .
Now, ’s objective is augmented as:
(7) 
zenke2017 explores a method similar to EWC for constraining the movement of salient parameters. Unlike in EWC, the computation of parameter importance in zenke2017 is performed online and does not require the betweentask phase for estimating Fisher information. This approach could alternatively be used here.
5 Experimental Results
We evaluate our approach with classconditional GANs, in a setting where new image classes are observed over time. Here, the classes are digit categories from the MNIST mnist and SVHN svhn datasets, and the conditional input
is a onehot encoding of the class label.
Unlike the typical concurrent training regime of a classconditional GAN, in the continual learning setup the distinct conditional distributions are encountered sequentially, where previously observed data is unavailable. While the proposed framework assumes that the same generator network is used to model each new distribution, it does not require that the same discriminator is used. Here, we experiment in the extreme setting where, similar to previously observed data, all previously trained discriminators are assumed inaccessible.
Architecturally, this setup may either be interpreted as new elements being appended to during each training session, with the corresponding initialization of new connections, or as some element of simply being always zero when the corresponding class is not part of the current task. We use the Adam optimization algorithm adam and reinitialize the optimizer parameters for each task. Thus, no updates occur for weights with zero gradient, making the two implementations equivalent.
On MNIST, we test our approach with a simple multilayer perceptron (MLP) GAN, while on SVHN we use a deep convolutional GAN (DCGAN, radford2015 ).
5.1 Mlp + Mnist
For MNIST, we first employ a simple GAN architecture where both and
are twolayer MLPs. The hidden layers of both networks are of size 128 with ReLU activation. In
, the output layer is of size 784 with sigmoid activation, corresponding to the generated image.has a single output with sigmoid activation, corresponding to the estimated probability that the input example was drawn from
.5.1.1 Overcoming forgetting
In Figure 2, we see the samples produced by a generator first trained concurrently on the digits 1 and 2. When digit 3 is encountered in the next training session and the generator’s objective is augmented with EWC, the previous distributions are not forgotten. This is in contrast to the catastrophic forgetting evident in Figure 1, where the generator trains on digit 3 with the standard objective. In Figure 4, we see samples from a generator first trained concurrently on digits 0 and 1, and then trained sequentially on the digits 2 through 6 with the augmented objective. Every displayed image was sampled after the final training session (on digit 6), and it is clear the previously learned distributions are not forgotten.
While catastrophic forgetting is significantly reduced when using the augmented objective, a small amount of peripheral noise is evident in the generated samples (e.g., in the digit 1 images in Figure 2. In general, the discriminative capability of tends to be local to and . That is, only learns to distinguish between real examples and the most recently generated samples. Images which are neither realistic nor similar to the current generated distribution are not meaningfully classified by .
This is evident when visualizing the saliency of ’s output layer parameters (Figure 3). The most critical parameters for fooling are those corresponding to the structure of the digit. Once matures enough during training such that peripheral noise is no longer produced, does not need to devote resources to detecting noise pixels. This limitation could potentially be addressed by training with experience replay, or by adding noise to some proportion of fake images is trained on. In practice, we find that higher capacity networks are less susceptible to this limitation.
Figure 5 shows the deformation to an image sampled from a previously learned distribution at a fixed
during the first epoch of training on a new class, with and without
’s EWCaugmented objective. While training on the digit 3 with the standard objective, forgets how to produce visually coherent images of the digit 1. The digit 1 image transitions to a skinny 3 as training progresses. No deformation is apparent when the augmented objective is used.5.2 Dcgan + Svhn
For SVHN, we use a conditional DCGAN architecture with three convolutional layers and one fully connected layer in both and . We append to in the input to , while in
a trainable linear transformation of
is added to the output of the first convolutional layer.5.2.1 Overcoming forgetting
Figure 6 shows outputs from (a) a DCGAN model trained on digits zero to four concurrently, (b) the same model after continuing training with digits five to nine, and (c) the same model after continuing training with the augmented loss on digits five to nine. Forgetting is clearly present in (b). In (c), the visual quality of the digits zero to four is similar to the quality of those digits in (a), while the visual quality of the digits five to nine is similar to the quality of those digits in (b).
5.3 Invariance to
In both MNIST + MLP and SVHN + DCGAN, we observed that the results of applying EWC were largely invariant to the magnitude of . Our reported results for MNIST used , while reported results for SVHN used . For SVHN we also trained models with and and observed very little difference in visual quality and no difference in diversity in the final results. An argument could be made for being the most principled parameter when interpreting EWC as a Taylor expansion of the previous task’s loss. However, with small values of the generator parameters tended to rapidly deviate from their previous task values at the beginning of training on a new task, resulting in poor quality samples, before gradually being pulled back and visual quality increasing over the course of training. With high values of we observed no loss in visual fidelity when beginning training on a new task, with rapid convergence to a model that produced good samples for all classes.
6 Conclusion
We have introduced an approach for continual learning with generative adversarial nets. Experimental results demonstrate that sequential training on different sets of conditional inputs utilizing an EWCaugmented loss counteracts catastrophic forgetting of previously learned distributions. The approach is general and applicable to any setting where the observed distribution of conditional inputs (e.g., class label, partially complete sample) changes over time, or where a conditional input representing the time of data capture can be appended to the data.
The promise of deep generative modeling is the ability to learn a model of the world from vast sources of data; to fully realize its potential, it will be necessary to learn from data at the point of capture rather than storing all of it for future use. This work is one step towards this goal.
Acknowledgments
This work was supported in part by the Department of Defense through the National Defense Science & Engineering Graduate Fellowship (NDSEG) Program.
References
 [1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16. ACM, 2016.
 [2] Ian Goodfellow, Jean PougetAbadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27. 2014.
 [3] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015.
 [4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimizatio. In ICLR, 2015.
 [5] Diederik P. Kingma and Max Welling. Autoencoding variational bayes. CoRR, abs/1312.6114, 2013.
 [6] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka GrabskaBarwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796, 2016.

[7]
Yann LeCun, Corinna Cortes, and Christopher JC Burges.
The mnist database of handwritten digits, 1998.
 [8] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1990.
 [9] Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, 2016.
 [10] James Martens. Secondorder optimization for neural networks. PhD Thesis, University of Toronto, 2016.
 [11] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24. Academic Press, 1989.

[12]
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera
y Arcas.
Communicationefficient learning of deep networks from decentralized
data.
In
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)
, 2016.  [13] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/1411.1784, 2014.
 [14] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
 [15] Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semisupervised knowledge transfer for deep learning from private training data. 2017.
 [16] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2015.
 [17] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1606.04671, 2016.
 [18] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems 28. 2015.
 [19] Friedemann Zenke, Ben Poole, and Surya Ganguli. Improved multitask learning through synaptic intelligence. CoRR, abs/1703.04200, 2017.