exploit the discriminative power of deep neural networks for the task of generative modeling. A GAN consists of two models: agenerator and a discriminator. The generator maps samples from a low-dimensional latent space onto the space of images. The discriminator tries to distinguish between images produced by the generator and real images. To coerce the generator to produce images that match the distribution of real images, we optimize it to fool the discriminator. It has been shown that various GAN minimax objectives are equivalent to minimizing corresponding divergences between the real and generated data.
Over the last several years, many researchers have successfully applied variants of GANs 
to tasks including synthesizing 2D images, face processing, image completion, image editing, 3D objects generation and reconstruction, fashion image generation, image and video super-resolutions, video generation, image-to-image translation, text-to-image generation, audio and text synthesis.
While generating images can be useful, we often want to infer latent representations given images. As an example, the latent code recovery methods is also used for generating fashion images that have horizontal symmetry . It is known that vectors that are close in latent space, generate visually similar images. Algebraic operations in latent vector space often lead to meaningful corresponding operations in image space.
However, the original GAN formulation gives no out-of-the-box method to reverse the mapping, projecting images back into latent space. How best to perform the reverse mapping (from image space to latent space) remains an open research problem. Authors in  suggests an extension to GAN in which a third model explicitly learns the reverse mapping.  suggest that inverting the generator is difficult, noting that, in principle, a single image may map to multiple latent vectors . They propose a gradient-based approach to recover latent vectors and evaluate the process on the reconstruction error in image space.
Recently,  proposes to recover latent vectors using a gradient-based method using “stochastic clipping”, and achieve successful recovery
of time given a certain residual threshold. The idea of “stochastic clipping” is based on the notion that the latent vectors have close to zero probability of landing on the boundary values.
In this paper, we show that latent code recovery can be used to denoise and deblur images, building on the method of . First, we show that for corrupted versions of images that are actually generated by our trained GAN, we can significantly denoise them, achieving higher quality as measured by PSNR, compared to BM3D , one of the best signal processing-based denoising methods. Next, we demonstrate that deblurring can be treated as an attribute in latent space, given the noise variance. Arithmetic on deblurring latent vector space denoises and deblurs the images even further. Finally, we show that even for unseen images, our method appears to denoise better than BM3D (Fig 1). To our knowledge, this is the first empirical demonstration that recovery of latent vectors in DCGAN can be used for image denoising and deblurring. After adding even significant amounts of Gaussian noise to images, we denoise the images with higher fidelity comparing with state-of-the-art denoising method.
2 Related Work
Image denoising is an actively researched inverse problem in low-level vision for last couple of decades. It has rich literature in traditional signal processing based methods. See  for a detailed survey. BM3D  is one of the best methods on image denoising in that domain. Recently, many researchers have proposed discriminative learning based methods for image denoising. Typically, these methods learn the image prior models and corresponding denoising function using CNNs. See [10, 11, 12] for overviews on these methods.
In this paper, we focus on the use of GANs for denoising and deblurring. Specifically, we project noisy images onto the range of the GAN by attempting to recover the latent vector which corresponds to the closest point on the GAN manifold. Several papers attempt gradient-based methods for inverting deep neural networks. Authors in 
invert discriminative CNNs for the purpose of interpreting hidden representations. To invert generative models, and  both optimize over latent vectors to minimize distance in image space, but neither reported that the inference procedure could faithfully recover ground truth latent vectors or that the inferred vectors (across multiple runs) tended to be proximal in latent space. In a different approach, , and , learn separate neural network encoders for performing the reverse mapping. The latent code recovery methods [7, 16] is also used for generating fashion images that have horizontal symmetry . Authors in  also note that the reverse projection can be used to remove Gaussian noise. However, they do not investigate any potential for latent vector arithmetic for deblurring. Authors in  allow the model to change the input vector that leads to better images according to the discriminator.
Recovery from an image generated by the generator, and from a real image are different. Former one can be considered as an inverse operation of an existing forward operation. However, attempting to recover the latent vector from a real image is more like a projection of it onto the manifold learned by the generator.
Our approach of recovering the latent vectors is based on . In order to recover the latent vector of a generated image, we produce an image for a latent vector . We then initialize a new, random vector of the same shape as . This new vector maps to a corresponding image . In order to reverse engineer the input , we successively update the components of in order to push the representation closer to the original image . In our experiments we minimize the norm, yielding the following optimization problem:
We optimize over by gradient descent, performing the update until some convergence criteria is met. Only solving for this optimization problem refers to as no clipping.
All latent vectors are sampled uniformly from the hyper-cube. To enforce this constraint, we apply the modified optimization
For projected gradient, we replace components that are too large with the maximum allowed value and components that are too small with the minimum allowed value. The authors in 
introduce a heuristic technique calledstochastic clipping. When using stochastic clipping, instead of setting components to or , we reassign the clipped components uniformly at random in the allowed range.
For the recovery of a real image the forward mapping does not necessarily exist. We perform the following optimization in this case.
with stochastic clipping, where denotes real images. We refer this latent vectors recovery methods as LVR. The generator in our experiments is trained with CelebA dataset . Example real images from CelebA dataset and their recovery using the LVR are shown in Fig 2. Additionally, some images from LFW  (the generator never saw these images) and their recovery are shown in Fig 3.
Authors in  note that the reverse projection can be used to remove Gaussian noise. In order to explore the denoising potential within this framework, we corrupt the real images with varying levels of Gaussian noise variance, and apply LVR for , where denotes real images and denotes Gaussian noise variance. The above denoising methods do not require the noise variance a priori unlike traditional denoising methods, and still produce better denoising results.
In case of high noise variance, the recovered images using LVR methods appear to be blurred. In order to explore the deblurring potential, we apply LVR on generated images with different noise variances represented by () and recover the corresponding latent vectors for those noise levels. Here, indexes over to and indexes over to . is the difference between average over sample and the average of their corresponding recovered for noise level . We observe that, adding to recovered latent vectors from other generated images with noise variance sharpens the recovered images. Interestingly, this happens while denoising real images. Empirically, there seems to exist vectors in latent space that can add sharpness in image space. If we denote the sharpness attribute in latent vector space as for noise variance , interestingly, we find the quality of is higher than for recovered latent vectors with LVR.
4 Experimental Results
We now summarize our experimental findings. All experiments are conducted with DCGANs as described by 
and re-implemented in Tensorflow by. We made necessary changes on top of . We train the generator using the CelebA  dataset.
|PSNR of reconstructed images with |
|No clipping||Projected Gradient||Stochastic Clipping|
We first perform the experiments for denoising of generated images after adding Gaussian noise with variance of pixels units by applying different strategies as described in . Stochastic clipping method outperforms projected gradient which performs better than no clipping strategy. Stochastic clipping method obtains more than dB higher PSNR than the without clipping strategy. Figure 4 shows the corresponding visual results.
We notice that the denoised images are significantly smoother than the images with which the generator was trained on. In order to improve the denoising results further, we explore the latent space for additional sharpness. We compare the latent space sharpness attribute-based method as LVR-SA, and always use the baseline of latent vector recovery with stochastic clipping.
As described in section 3, we first generate a set of images, , corrupt them with Gaussian noise with different variance. For each noise variance, we recover the latent vectors using stochastic clipping strategy. We then take the difference between average latent vectors that generated the image and the average latent vectors that were recovered. This difference serves as sharpness attribute. First, we add this sharpness attribute to the recovered vectors for those images. We observe that after adding the sharpness attribute, the PSNR increases upto dB. Next, we apply the same sharpness attribute while recovering a different set of real images, . Interestingly, the same sharpness attribute helps increase the PSNR by more than dB. Table 2 summarizes the above results, where the first row corresponds to the former case and the second row corresponds to the later scenario respectively. Figure 5 shows the corresponding visualizations. For each column, the top images are the original images. We add Gaussian noise to those images and recover them in the middle row with stochastic clipping. Third row denotes the recovered images after adding the sharpness attribute . The generator is trained only on celebA . In the left column, the images are sampled from CelebA dataset and in the right, the images are sampled from LFW  dataset.
|Effect of Sharpness attribute|
|same image sets|
|different image sets|
Finally we compare the denoising results utilizing the above sharpness attribute method with stochastic clipping LVR and BM3D. Table 3 shows the results for different noise variances. The RGB images use -bit representation. The pixels can take values in the range of both inclusive for R,G, and B space. We experiment with high noise variance values. For low noise variance, the recovered images with different methods yield similar quality image deniosing. Latent vector recovery (LVR) with stochastic clipping yields better denoising results. This LVR method does not require the noise variance as a prior knowledge. Moreover, adding the sharpness latent vector (LVR-SA) improves the image quality on top of LVR. BM3D and LVR-SA both require a-priori knowledge of the noise variance. Figure 6 demonstrates the corresponding visual results. Denoising with the proposed method recovers clearly better quality images than BM3D.
We show that latent vector recovery from GAN, in practice, can be used for image denoising. The denoised image quality is superior to methods such as BM3D. Additionally, we show that sharpness can be treated as an attribute in latent vector space. Adding this sharpness attribute leads to even higher quality image denoising than simple latent vector recovery based methods.
-  Ian et al. Goodfellow, “Generative adversarial nets,” in Advances in neural information processing systems, 2014.
-  Alec Radford, Luke Metz, and Soumith Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434, 2015.
-  Avinash Hindupur, “The GAN Zoo,” https://deephunt.in/the-gan-zoo-79597dc8c347, 2016.
-  Vishnu Makkapati and Arun Patro, Enhancing Symmetry in GAN Generated Fashion Images, pp. 405–410, Springer International Publishing, Cham, 2017.
-  Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell, “Adversarial feature learning,” arXiv preprint arXiv:1605.09782, 2016.
-  Antonia Creswell and Anil Anthony Bharath, “Inverting the generator of a generative adversarial network,” arXiv preprint arXiv:1611.05644, 2016.
-  Zachary C. Lipton and Subarna Tripathi, “Precise recovery of latent vectors from generative adversarial networks,” CoRR, vol. abs/1702.04782, 2017.
-  K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” Trans. Img. Proc., vol. 16, no. 8, pp. 2080–2095, Aug. 2007.
-  P. Milanfar, “A tour of modern image filtering: New insights and methods, both practical and theoretical,” IEEE Signal Processing Magazine, vol. 30, no. 1, pp. 106–128, Jan 2013.
-  Kai Zhang, Wangmeng Zuo, and Lei Zhang, “Ffdnet: Toward a fast and flexible solution for CNN based image denoising,” CoRR, vol. abs/1710.04026, 2017.
-  Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising,” CoRR, vol. abs/1608.03981, 2016.
-  Nithish Divakar and R. Venkatesh Babu, “Image denoising via cnns: An adversarial approach,” CoRR, vol. abs/1708.00159, 2017.
-  Aravindh Mahendran and Andrea Vedaldi, “Understanding deep image representations by inverting them,” in , 2015, pp. 5188–5196.
-  Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein, “Unrolled generative adversarial networks,” arXiv preprint arXiv:1611.02163, 2016.
-  Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville, “Adversarially learned inference,” in ICLR, 2017.
-  Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G. Dimakis, “Compressed sensing using generative models,” arXiv preprint arXiv:1703.03208, 2017.
-  Alexander B. Jung, “Learning to avoid errors in gans by manipulating input spaces,” arXiv preprint arXiv:1707.00768, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang,
“Deep learning face attributes in the wild,”in Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller,
“Labeled faces in the wild: A database for studying face recognition in unconstrained environments,”Tech. Rep. 07-49, University of Massachusetts, Amherst, October 2007.
-  Brandon Amos, “Image Completion with Deep Learning in TensorFlow,” http://bamos.github.io/2016/08/09/deep-completion, 2016.