Semantic inpainting refers to filling up of missing pixels in a given image by leveraging neighborhood information. Traditional methods [1, 3] were mainly successful when deployed on background scenes and images with repeated textures. However, they fail to learn complex semantic representations and thereby manifest unpleasing reconstructions on complex non-repetitive textured objects. With the advent of Generative Adversarial Networks (GAN), there has been a recent surge of interest [13, 4, 14, 7] to solve inpainting with deep generative models. There are mainly two schools of approach:
Fully unsupervised: This approach, first proposed by Yeh et al.  aligns with the concept of the pioneering paper of GAN . In , the objective is learn a GAN model to generate realistic images conditioned on noise priors only and inpaiting is done by iteratively matching a masked/damaged image to its ‘best matching’ noise prior. This method does not require any paired training set (masked, unmasked) and hence we term it as ‘unsupervised’.
Hybrid: These methods [14, 7, 10, 4] in general rely on initial training with usual reconstruction loss on a paired (masked, unmasked) dataset. Since loss based reconstructions manifest lack of high frequency components, the next step is to push the solutions nearer to original data manifold with an additional adversarial loss. It is to be noted, that without the initial supervised training phase, these methods fail to work and thus we term this framework as ‘hybrid’ approach.
Motivation: For , being fully unsupervised comes at a cost of significant inference time due to an iterative search for a matching noise prior. Hybrid methods perform test time inference in one single forward pass and thus research has been dedicated mainly towards this genre. However, going against the trend, we advocate the former method because true potential of GAN is appreciated when there is no source of supervision. The motivation in this paper is to primarily reduce the inference run time of , yet achieve better/similar reconstruction performance compared to . Towards this, the paper presents the following contributions:
1. A better initializing method for the iterative optimization of  to speedup inference run time on single image inpainting by 4.5-5.
2. First demonstration of totally unsupervised GAN based inpainting on videos (in context of error concealment) with speedup upto 80 by leveraging temporal redundancy
3. A group consistency loss for a more temporally consistent sequence reconstruction and thereby leading to more pleasing spatio-temporal experience as ascertained by the MOVIE metric 
4. Exhaustive experiments on SVHN, Standford Cars, CelebA image dataset and VidTIMID video dataset manifest the benefits of our approach
2 GAN preliminaries
A GAN model consists of two deep neural nets, viz., generator, , and discriminator, . The task of the generator is to create an image,
with a noise prior vector,, as input. is sampled from a known distribution, ; usually . The discriminator has to distinguish between real samples(sampled from real distribution, ) and generated samples. The game is played on :
We build upon the unsupervised inpainting framework of Yeh et al.. Given a masked image, , corresponding to an original image, , and a pre-trained GAN model, the idea is to iteratively find the ‘closest’ vector (starting randomly from ) which results in a reconstructed image whose semantics are similar to corrupted image. is optimized as,
where is the binary mask with zeros on masked region else unity, is the pointwise multiplier and is the objective function to be minimized. Interesting to note is that the objective function never assumes knowledge of pixel intensities inside the masked region(and thus the term ‘unsupervised’). Upon convergence, the inpainted image, , is given as, . The objective function, is composed of two components:
Fidelity Loss: This loss ensures that the predicted noise prior preserves fidelity between generated image and the original unmasked regions.
Perceptual Loss: This loss ensures that the inpainted output lies near the original/real data manifold and is measured by the log likelihood of real class assigned by the pre-trained discriminator;
The overall objective, , where controls the relative importance of .
3.1 Better initiation for noise prior search
One of the fundamental drawbacks of Yeh et al. is the iterative optimization requirement of Eq.2. A random usually tends to generate images quite disparate from the concerned maksed image and thus the optimization requires multiple updates of . In fact, the authors in
 suggest around 1000 rounds of iterations per image. Our motivation is to initiate by respecting some global statistics of the concerned masked image.
Nearest neighbor search: After training a GAN, we store (one time offline task) a pool, , of images by passing random noise vectors through the pre-trained generator. For a given damaged image, , we perform a nearest neighbor search over the pool, , to identify ‘the closest’ matching pair. Specifically, we perform the matching between and a candidate pooled image (generated from ), , based on a distance metric, . Please note, even during matching we are not exploiting the masked region of . While formulating we want to make sure that we not only match the overall color statistics of the damaged image but also respect the overall structure. Thus has got two components:
Data loss: This loss penalizes if the pixel intensities of a pooled image, , deviate from the damaged image, ;
Structure loss: This loss penalizes if the structure(captured in essence with gradients) of pooled image deviates from damaged image. Structure loss, is defined as:
where and are horizontal and vertical gradient operators. The final matching criterion is, , where controls relative importance of . Effect of is discussed in Fig.1. The initial noise vector, is given by,
3.2 Video Inpainting: Exploiting temporal redundancy
To our best knowledge, this is the first demonstration of unsupervised GAN based inpainting on videos. By video inpainting we refer to concept of error concealment in video, i.e., to recover damaged/masked portion of a frame. A naive application of  would be to apply single image model independently on each frame. This poses two problems, viz, a) such approach does not leverage temporal redundancy among neighboring frames b) independent frame level reconstructions result in temporal inconsistency in a sequence. We propose to address these challenges with two innovations.
Reuse of predicted vector: It is safe to assume that neighboring frames are coherent in appearance and thus the noise priors. Thus, it makes sense to initiate . This drastically speeds up optimization by almost 80 compared to vanilla version of Yeh et al.. We refer to this proposed method as Proposed (Re) in all experiments.
Group consistency loss: Even though we initialize with , the final solution for time step () can diverge away appreciably from . This would mean that there will abrupt changes in scene appearance when the frames are viewed as a sequence. To enforce smooth temporal dynamics we impose a group consistency loss, , by constraining a group of reconstructed frames to be similar. Disparity between two generated images can be expressed with corresponding disparity between the corresponding vectors . Specifically, we impose consistency loss over a window of neighboring frames,
Please note that is imposed only over a neighboring window of frames and not over entire video. Combination of this loss + reuse of vector is denoted as Proposed (Re + G) in experiments.
4 Experiment Settings
Datasets: For image inpainting we tested on SVHN, Standford Cars @ 6464 resolution and CelebA @ 6464 and 128128 resolution. For video inpainting, we experimented at 128128 resolution on VidTIMIT dataset.
Network Architectures: For fair comparison with our baseline of , we borrowed their architectures for both generator and discriminator and followed their paradigm of training GAN. Parameter, , was set to 0.01 following .
Balacing data loss and structure loss:Hyperparameter, sets the relative importance of structure loss, , over data loss . Setting means initial solution will not explicitly preserve edge information and just retrieve nearest image based on raw intensity. On contrary, a high value of will enforce only edge preservation without respecting the intensity. See Fig.1 for illustration of these two extreme cases. In either of these cases, initial solution can be appreciably different than desired solution and will thus require longer iterative refinements. We reserved a validation dataset on which we test . gave peak speedup across datasets and thus is set to 0.01 for all experiments. See Fig.2 for some examples of initial solutions with our proposed method v/s random initialization of .
Pool size () for nearest neighbor search: Nearest neighbor search with an ideal generator () will be able to retrieve the exact vector corresponding to a masked image, , if we allow, . However, for practical viability it is not possible to search over every possible vector. On validation set, we experiment with different and compare the average MS-SSIM between initial masked solution and masked image. is set to 300(on all datasets unless otherwise stated), above which, the MS-SSIM does not increase appreciably.
Selecting group consistency window, : Setting (in Eq.8) to a large value results in over smoothing of a sequence in temporal dimension and thus leads to degraded MOVIE metric due to poor perceptual quality. , on the other hand will have no effect in incorporating temporal coherence which again hurts MOVIE metric. On held out validation set of VidTIMIT we experimented with . was selected which, on average, yielded best MOVIE metric. So, for a sequence, every alternate 5 frame (pivot frame) is inpainted as a single image. Intermediate frames are initiated with inpainted pivot frame and solutions of a group are additionally constrained by .
Comparing Methods: Our primary comparing method is the unsupervised baseline of . However, for the completeness of the paper we also provide comparisons with hybrid frameworks of [10, 4, 14]. Please note, the later approaches are only successful if trained initially with supervised reconstruction loss. If trained with only unsupervised adversarial loss, these method fail drastically.
|Yeh et al.||9.0||33.5|
|Proposed (Re + G)||-||0.41|
|||||||||Proposed (Re)||Proposed (Re + G)|
Speedup in optimization: With respect to our unsupervised baseline, on average, we achieved about 5 speedup for single image inpainting. On videos the speedup is almost 80. See Table 2 for speed comparisons.
Image Inpainting: In Fig. 3 we show some exemplary inpainting comparison with . Even with appreciable speedup we usually achieve better (or similar visual performance) to . We also show some visual comparison with recent hybrid benchmarks in Fig.4. Recently [6, 13, 14]
researchers have shown that PSNR metric is not fully justifiable to assess tasks such as inpainting and super resolution. However, for reference we also report PSNR in TableLABEL:table_psnr.
Pseudo sequences and temporal consistency Before analyzing effects of our proposed losses on real videos, we study the effects on a simpler case of pseudo sequences. A pseudo sequence of length, is basically a single image replicated times but masked with different masks. An ideal model would inpaint all the frames identically. We can define temporal consistency, as, combinations of . In Table LABEL:tab_consistency we report temporal consistency over different datasets. It can be seen that proposed initialization technique of improves consistency with further improvement brought by group consistency loss. Even the current hybrid benchmarks [14, 10, 4] manifest greater inconsistency because these methods do not leverage any temporal information. Exemplary visualizations are shown in Fig. 5. Note, gives an indication of temporal consistency only (which can be studied only on such pseudo sequences). It does not give essence of spatial correctness. For example, a model might reconstruct all blank images which will give high , but will yield worse MOVIE metric.
Analyzing real video inpainting performance PSNR and MS-SSIM are not well suited for judging a reconstructed video since these metrics are agnostic to temporal dimension. We advocate using the MOVIE metric which considers spatial, temporal and spatio-temporal aspects by comparing original and reconstructed sequence. In Table 4 we report average MOVIE metric on VidTIMIT test set. Our proposed modifications to  significantly improves reconstructed video quality.
In this paper, we first discussed the problem of impractical long inference time of the recent completely unsupervised inpainting framework of . We then proposed to speedup the iterative optimization of  by better initialization technique on images and also leveraging temporal redundancy in videos. In the process, we achieved a significant speedup. Several comparisons were also done with current hybrid benchmarks and we achieved comparable performance. Future work might consider replacing the iterative optimization by learning to project a damaged image into noise prior space.
-  (2009) PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (ToG) 28 (3), pp. 24. Cited by: §1.
-  (2014) Generative adversarial nets. In NIPS, pp. 2672–2680. Cited by: §1.
-  (2007) Scene completion using millions of photographs. In ACM Transactions on Graphics (TOG), Vol. 26, pp. 4. Cited by: §1.
-  (2017) Globally and locally consistent image completion. ACM Transactions on Graphics (TOG) 36 (4), pp. 107. Cited by: §1, Table 1, Table 3, Table 4, §4, §5.
-  (2013) 3D object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia. Cited by: §4.
-  (2017) Photo-realistic single image super-resolution using a generative adversarial network.. In CVPR, Vol. 2, pp. 4. Cited by: §5.
-  (2017) Generative face completion. In , Vol. 1, pp. 3. Cited by: §1.
-  (2015) Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738. Cited by: §4.
-  (2011) Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, Vol. 2011, pp. 5. Cited by: §4.
-  (2016) Context encoders: feature learning by inpainting. In CVPR, pp. 2536–2544. Cited by: §1, Table 1, Table 3, Table 4, §4, Figure 4, §5.
-  (2009) Multi-region probabilistic histograms for robust and scalable identity inference. In International Conference on Biometrics, pp. 199–208. Cited by: §4.
-  (2010) Motion tuned spatio-temporal quality assessment of natural videos. IEEE transactions on image processing 19 (2), pp. 335–350. Cited by: §1, Table 4, §4, §5.
-  (2017) Semantic image inpainting with deep generative models. In CVPR, pp. 5485–5493. Cited by: §1, §1, §3.1, §3.2, §3, Figure 2, Table 1, Table 2, Table 3, Table 4, §4, Figure 3, Figure 5, §5, §6.
-  (2018) Generative image inpainting with contextual attention. In CVPR, Cited by: §1, Table 1, Table 3, Table 4, §4, Figure 4, §5.
-  (2016) Generative visual manipulation on the natural image manifold. In ECCV, pp. 597–613. Cited by: §3.2.