A list of popular github projects related to deep learning
Over the last few years, deep learning techniques have yielded significant improvements in image inpainting. However, many of these techniques fail to reconstruct reasonable structures as they are commonly over-smoothed and/or blurry. This paper develops a new approach for image inpainting that does a better job of reproducing filled regions exhibiting fine details. We propose a two-stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network. The edge generator hallucinates edges of the missing region (both regular and irregular) of the image, and the image completion network fills in the missing regions using hallucinated edges as a priori. We evaluate our model end-to-end over the publicly available datasets CelebA, Places2, and Paris StreetView, and show that it outperforms current state-of-the-art techniques quantitatively and qualitatively.READ FULL TEXT VIEW PDF
A list of popular github projects related to deep learning
EdgeConnect: Structure Guided Image Inpainting using Edge Prediction, ICCV 2019 https://arxiv.org/abs/1901.00212
EdgeConnect for iOS implemented using CoreML.
An AI based web app to translate the text on your images while keeping the background of the image same as original.
An image-inpainting application forked from EdgeConnect
Image inpainting, or image completion, involves filling in missing regions of an image. It is an important step in many image editing tasks. It can, for example, be used to fill in the holes left after removing unwanted objects from an image. Humans have an uncanny ability to zero in on visual inconsistencies. Consequently, the filled regions must be perceptually plausible. Among other things, the lack of fine structure in the filled region is a giveaway that something is amiss, especially when the rest of the image contain sharp details. The work presented in this paper is motivated by our observation that many existing image inpainting techniques generate over-smoothed and/or blurry regions, failing to reproduce fine details. We divide image inpainting into a two-stage process (Figure 1
): edge generation and image completion. Edge generation is solely focused on hallucinating edges in the missing regions. The image completion network uses the hallucinated edges and estimates RGB pixel intensities of the missing regions. Both stages follow an adversarial framework
to ensure that the hallucinated edges and the RGB pixel intensities are visually consistent. Both networks incorporate losses based on deep features to enforce perceptually realistic results. Like most computer vision problems, image inpainting predates the wide-spread use of deep learning techniques. Broadly speaking, traditional approaches for image inpainting can be divided into two groups: diffusion-based and patch-based. Diffusion-based methods propagate background data into the missing region by following a diffusive process typically modeled using differential operators[4, 14, 27, 2]. Patch-based methods, on the other hand, fill in missing regions with patches from a collection of source images that maximize patch similarity [7, 21]. These methods, however, do a poor job of reconstructing complex details that may be local to the missing region. More recently deep learning approaches have found remarkable success at the task of image inpainting. These schemes fill the missing pixels using learned data distribution. They are able to generate coherent structures in the missing regions, a feat that was nearly impossible for traditional techniques. While these approaches are able to generate missing regions with meaningful structures, the generated regions are often blurry or suffer from artifacts, suggesting that these methods struggle to reconstruct high frequency information accurately. Then, how does one force an image inpainting network to generate fine details? Since image structure is well-represented in its edge mask, we show that it is possible to generate superior results by conditioning an image inpainting network on edges in the missing regions. Clearly, we do not have access to edges in the missing regions. Rather, we train an edge generator that hallucinates edges in these areas. Our approach of “lines first, color next” is partly inspired by our understanding of how artists work . “In line drawing, the lines not only delineate and define spaces and shapes; they also play a vital role in the composition”, says Betty Edwards, highlights the importance of sketches from an artistic viewpoint . Edge recovery, we suppose, is an easier task than image completion. Our proposed model essentially decouples the recovery of high and low-frequency information of the inpainted region. We evaluate our proposed model on standard datasets CelebA , Places2 , and Paris StreetView . We compare the performance of our model against current state-of-the-art schemes. Furthermore, we provide results of experiments carried out to study the effects of edge information on the image inpainting task. Our paper makes the following contributions:
An edge generator capable of hallucinating edges in missing regions given edges and grayscale pixel intensities of the rest of the image.
An image completion network that combines edges in the missing regions with color and texture information of the rest of the image to fill the missing regions.
An end-to-end trainable network that combines edge generation and image completion to fill in missing regions exhibiting fine details.
We show that our model can be used in some common image editing applications, such as object removal and scene generation. Our source code is available at:
Diffusion-based methods propagate neighboring information into the missing regions [4, 2].  adapted the Mumford-Shah segmentation model for image inpainting by introducing Euler’s Elastica. However, reconstruction is restricted to locally available information for these diffusion-based methods, and these methods fail to recover meaningful structures in the missing regions. These methods also cannot adequately deal with large missing regions. Patch-based methods fill in missing regions (, targets) by copying information from similar regions (, sources) of the same image (or a collection of images). Source regions are often blended into the target regions to minimize discontinuities [7, 21]. These methods are computationally expensive since similarity scores must be computed for every target-source pair. PatchMatch  addressed this issue by using a fast nearest neighbor field algorithm. These methods, however, assume that the texture of the inpainted region can be found elsewhere in the image. This assumption does not always hold. Consequently, these methods excel at recovering highly patterned regions such as background completion but struggle at reconstructing patterns that are locally unique. One of the first deep learning methods designed for image inpainting is context encoder , which uses an encoder-decoder architecture. The encoder maps an image with missing regions to a low-dimensional feature space, which the decoder uses to construct the output image. However, the recovered regions of the output image often contain visual artifacts and exhibit blurriness due to the information bottleneck in the channel-wise fully connected layer. This was addressed by Iizuka  by reducing the number of downsampling layers, and replacing the channel-wise fully connected layer with a series of dilated convolution layers . The reduction of downsampling layers are compensated by using varying dilation factors. However, training time was increased significantly111Model by  required two months of training over four GPUs. due to extremely sparse filters created using large dilation factors. Yang  uses a pre-trained VGG network  to improve the output of the context-encoder, by minimizing the feature difference of image background. This approach requires solving a multi-scale optimization problem iteratively, which noticeably increases computational cost during inference time. Liu  introduced “partial convolution” for image inpainting, where convolution weights are normalized by the mask area of the window that the convolution filter currently resides over. This effectively prevents the convolution filters from capturing too many zeros when they traverse over the incomplete region. Recently, several methods were introduced by providing additional information prior to inpainting. Yeh  trains a GAN for image inpainting with uncorrupted data. During inference, back-propagation is employed for iterations to find the representation of the corrupted image on a uniform noise distribution. However, the model is slow during inference since back-propagation must be performed for every image it attempts to recover. Dolhansky and Ferrer  demonstrate the importance of exemplar information for inpainting. Their method is able to achieve both sharp and realistic inpainting results. Their method, however, is geared towards filling in missing eye regions in frontal human face images. It is highly specialized and does not generalize well. Contextual Attention  takes a two-step approach to the problem of image inpainting. First, it produces a coarse estimate of the missing region. Next, a refinement network sharpens the result using an attention mechanism by searching for a collection of background patches with the highest similarity to the coarse estimate.  takes a similar approach and introduces a “patch-swap” layer which replaces each patch inside the missing region with the most similar patch on the boundary. These schemes suffer from two limitations: 1) the refinement network assumes that the coarse estimate is reasonably accurate, and 2) these methods cannot handle missing regions with arbitrary shapes. Free-form inpainting method proposed in  is perhaps closest in spirit to our scheme. It uses hand-drawn sketches to guide the inpainting process. Our method does away with hand-drawn sketches and instead learns to hallucinate edges in the missing regions.
The inpainting technique proposed in this paper subsumes two disparate computer vision problems: Image-to-Edges and Edges-to-Image. There is a large body of literature that addresses “Image-to-Edges” problems [5, 10, 26, 29]. Canny edge detector, an early scheme for constructing edge maps, for example, is roughly 30 years old . Dollár and Zitnikc  use structured learning  on random decision forests to predict local edge masks. Holistically-nested Edge Detection (HED)  is a fully convolutional network that learns edge information based on its importance as a feature of the overall image. In our work, we train on edge maps computed using Canny edge detector. We explain this in detail in Section 4.1 and Section 5.3. Traditional “Edges-to-Image” methods typically follow a bag-of-words approach, where image content is constructed through a pre-defined set of keywords. These methods, however, are unable to accurately construct fine-grained details especially near object boundaries. Scribbler  is a learning-based model where images are generated using line sketches as the input. The results of their work possess an art-like quality, where color distribution of the generated result is guided by the use of color in the input sketch. Isola  proposed a conditional GAN framework 
, called pix2pix, for image-to-image translation problems. This scheme can use available edge information asa priori. CycleGAN  extends this framework and finds a reverse mapping back to the original data distribution. This approach yields superior results since the aim is to learn the inverse of the forward mapping.
We propose an image inpainting network that consists of two stages: 1) edge generator, and 2) image completion network (Figure 2). Both stages follow an adversarial model , each stage consists of a generator/discriminator pair. Let and be the generator and discriminator for the edge generator, and and be the generator and discriminator for the image completion network, respectively. To simplify notation, we will use these symbols also to represent the function mappings of their respective networks. Our generators follow an architecture similar to the method proposed by Johnson 
, which has achieved impressive results for style transfer, super-resolution[40, 17], and image-to-image translation . Specifically, the generators consist of encoders that down-sample twice, followed by eight residual blocks  and decoders that up-sample images back to the original size. Dilated convolutions with a dilation factor of two are used instead of regular convolutions in the residual layers, resulting in a receptive field of at the final residual block. For discriminators, we use a PatchGAN [23, 57] architecture, which determines whether or not overlapping image patches of size are real. We use instance normalization  across all layers of the network222The details of our architecture are in appendix A.
Let be ground truth images. Their edge map and grayscale counterpart will be denoted by and , respectively. In the edge generator, we use the masked grayscale image as the input, its edge map , and image mask as a pre-condition (1 for the missing region, 0 for background). Here, denotes the Hadamard product. The generator predicts the edge map for the masked region
We use and conditioned on as inputs of the discriminator that predicts whether or not an edge map is real. The network is trained with an objective comprised of an adversarial loss and feature-matching loss 
where and are regularization parameters. The adversarial loss is defined as
The feature-matching loss compares the activation maps in the intermediate layers of the discriminator. This stabilizes the training process by forcing the generator to produce results with representations that are similar to real images. This is similar to perceptual loss [24, 16, 15], where activation maps are compared with those from the pre-trained VGG network. However, since the VGG network is not trained to produce edge information, it fails to capture the result that we seek in the initial stage. The feature matching loss is defined as
where is the final convolution layer of the discriminator, is the number of elements in the ’th activation layer, and is the activation in the ’th layer of the discriminator. Spectral normalization (SN) 
further stabilizes training by scaling down weight matrices by their respective largest singular values, effectively restricting the Lipschitz constant of the network to one. Although this was originally proposed to be used only on the discriminator, recent works[54, 36] suggest that generator can also benefit from SN by suppressing sudden changes of parameter and gradient values. Therefore, we apply SN to both generator and discriminator. Spectral normalization was chosen over Wasserstein GAN (WGAN),  as we found that WGAN was several times slower in our early tests. Note that only is maximized over since is used to retrieve activation maps for . For our experiments, we choose and .
The image completion network uses the incomplete color image as input, conditioned using a composite edge map . The composite edge map is constructed by combining the background region of ground truth edges with generated edges in the corrupted region from the previous stage, . The network returns a color image , with missing regions filled in, that has the same resolution as the input image:
This is trained over a joint loss that consists of an loss, adversarial loss, perceptual loss, and style loss. To ensure proper scaling, the loss is normalized by the mask size. The adversarial loss is defined similar to Eq. 3, as
We include the two losses proposed in [16, 24] commonly known as perceptual loss and style loss . As the name suggests, penalizes results that are not perceptually similar to labels by defining a distance measure between activation maps of a pre-trained network. Perceptual loss is defined as
where is the activation map of the ’th layer of a pre-trained network. For our work, corresponds to activation maps from layers and
of the VGG-19 network pre-trained on the ImageNet dataset. These activation maps are also used to compute style loss which measures the differences between covariances of the activation maps. Given feature maps of sizes , style loss is computed by
where is a Gram matrix constructed from activation maps . We choose to use style loss as it was shown by Sajjadi  to be an effective tool to combat “checkerboard” artifacts caused by transpose convolution layers 
. Our overall loss function is
For our experiments, we choose , , and . We noticed that the training time increases significantly if spectral normalization is included. We believe this is due to the network becoming too restrictive with the increased number of terms in the loss function. Therefore we choose to exclude spectral normalization from the image completion network.
, we generate training labels (edge maps) using Canny edge detector. The sensitivity of Canny edge detector is controlled by the standard deviation of the Gaussian smoothing filter. For our tests, we empirically found that yields the best results (Figure 6). In Section 5.3, we investigate the effect of the quality of edge maps on the overall image completion. For our experiments, we use two types of image masks: regular and irregular. Regular masks are square masks of fixed size (25% of total image pixels) centered at a random location within the image. We obtain irregular masks from the work of Liu . Irregular masks are augmented by introducing four rotations (
) and a horizontal reflection for each mask. They are classified based on their sizes relative to the entire image in increments of(, 0-10%, 10-20%, ).
Our proposed model is implemented in PyTorch. The network is trained usingimages with a batch size of eight. The model is optimized using Adam optimizer  with and . Generators are trained separately using Canny edges with learning rate until the losses plateau. We lower the learning rate to and continue to train and until convergence. Finally, we fine-tune the networks by removing , then train and end-to-end with learning rate until convergence. Discriminators are trained with a learning rate one tenth of the generators’.
Our proposed model is evaluated on the datasets CelebA , Places2 , and Paris StreetView . Results are compared against the current state-of-the-art methods both qualitatively and quantitatively.
Figure 4 shows a sample of images generated by our model. For visualization purposes, we reverse the colors of . Our model is able to generate photo-realistic results with a large fraction of image structures remaining intact. Furthermore, by including style loss, the inpainted images lack any “checkerboard” artifacts in the generated results. As importantly, the inpainted images exhibit minimal blurriness.
Figure 3 compares images generated by our method with those generated by other state-of-the-art techniques. The images generated by our proposed model are closer to ground truth than images from other methods. We conjecture that when edge information is present, the network only needs to learn the color distribution, without having to worry about preserving image structure.
We measure the quality of our results using the following metrics: 1) relative ; 2) structural similarity index (SSIM) , with a window size of ; and 3) peak signal-to-noise ratio (PSNR). These metrics assume pixel-wise independence, which may assign favorable scores to perceptually inaccurate results. Therefore, we also include Fréchet Inception Distance (FID) . Recent works [55, 54, 9] have shown that metrics based on deep features are closer to those based on human perception. FID measures the Wasserstein-2 distance between the feature space representations of real and inpainted images using a pre-trained Inception-V3 model . The results over Places2 dataset are reported in Table 1. Note that these statistics are based on the synthesized image which mostly comprises of the ground truth image. Therefore our reported FID values are lower than other generative models reported in .
Figure 5 shows the performance of our model for various mask sizes. Statistics for competing techniques are obtained using their respective pre-trained weights, where available333https://github.com/JiahuiYu/generative_inpainting 444https://github.com/satoshiiizuka/siggraph2017_inpainting. Results for Partial Convolution (PConv)  are taken from their paper as the source code is not available at the time of writing. Note that (%) errors for PConv are lower than those achieved by our method and those reported in CA  and GLCIC . While we are not sure why this is so, we suspect PConv is computing this score differently than how we compute it. Our statistics are calculated over random images in the test set.
We evaluate our results using the human perceptual metrics 2 alternative forced choice (2AFC) and just noticeable differences (JND). For 2AFC, we ask participants whether or not an image is real from a pool of randomized images. For JND, we ask participants to select the more realistic image from pairs of real and generated images. Participants are given two seconds for each test. The tests are performed over 300 images for each model and mask size. Each image is shown 10 times in total. The results are summarized in Table 2.
|JND (%)||10-20%||21 1.2%||16.9 1.1%||39.7 1.5%|
|20-30%||15.5 1.1%||14.3 1%||37 1.5%|
|30-40%||12.9 1%||12.3 1%||27.5 1.3%|
|40-50%||12.7 2%||10.9 0.9%||25.4 1.3%|
|2AFC (%)||10-20%||38.7 1.8%||22.5 1.5%||88.7 1.2%|
|20-30%||23.4 1.5%||12.1 1.2%||77.6 1.5%|
|30-40%||13.5 1.3%||4.3 0.7%||66.4 1.8%|
|40-50%||9.9 1%||2.8 0.6%||58 1.8%|
We now turn our attention to the key assumption of this work: edge information helps with image inpainting. Table 3 shows inpainting results with and without edge information. Our model achieved better scores for every metric when edge information was incorporated into the inpainting model, even when a significant portion of the image is missing.
Next, we turn to a more interesting question: How much edge information is needed to see improvements in the generated images? We again use Canny edge detector to construct edge information. We use the parameter to control the amount of edge information available to the image completion network. Specifically, we train our image completion network using edge maps generated for , and we found that the best image inpainting results are obtained with edges corresponding to , across all datasets shown in Figure 6. For large values of , too few edges are available to make a difference in the quality of generated images. On the other hand, when is too small, too many edges are produced, which adversely affect the quality of the generated images. We used this study to set when creating ground truth edge maps for the training of the edge generator network.
Figure 7 shows how different values of affects the inpainting task. Note that in a region where edge data is sparse, the quality of the inpainted region degrades. For instance, in the generated image for , the left eye was reconstructed much sharper than the right eye.
We use Canny edge detector to produce training labels for the edge generator network due to its speed, robustness, and ease of use. Canny edges are one-pixel wide, and are represented as binary masks (1 for edge, 0 for background). Edges produced with HED , however, are of varying thickness, and pixels can have intensities ranging between 0 and 1. We noticed that it is possible to create edge maps that look eerily similar to human sketches by performing element-wise multiplication on Canny and HED edge maps (Figure 8). We trained our image completion network using the combined edge map. However, we did not notice any improvements in the inpainting results.555Further analysis with HED are available in appendix C.
We proposed EdgeConnect, a new deep learning model for image inpainting tasks. EdgeConnect comprises of an edge generator and an image completion network, both following an adversarial model. We demonstrate that edge information plays an important role in the task of image inpainting. Our method achieves state-of-the-art results on standard benchmarks, and is able to deal with images with multiple, irregularly shaped missing regions. The trained model can be used as an interactive image editing tool. We can, for example, manipulate objects in the edge domain and transform the edge maps back to generate a new image. This is demonstrated in Figure 10. Here we have removed the right-half of a given image to be used as input. The edge maps, however, are provided by a different image. The generated image seems to share characteristics of the two images. Figure 11 shows examples where we attempt to remove unwanted objects from existing images. We plan to investigate better edge detectors. While effectively delineating the edges is more useful than hundreds of detailed lines, our edge generating model sometimes fails to accurately depict the edges in highly textured areas, or when a large portion of the image is missing (as seen in Figure 9). We believe our fully convolutional generative model can be extended to very high-resolution inpainting applications with an improved edge generating system.
This research was supported in part by NSERC Discovery Grant. The authors would like to thank Konstantinos G. Derpanis for helpful discussions and feedbacks. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4380–4389, 2015.
Texture synthesis using convolutional neural networks.In Advances in Neural Information Processing Systems, pages 262–270, 2015.
Image-to-image translation with conditional adversarial networks.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Proceedings of the 35th International Conference on Machine Learning, 2018.
Places: A 10 million image database for scene recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
We follow a similar naming convention as those presented in . Let ck denote a
Convolution-SpectralNorm-InstanceNorm-ReLU layer with
filters and stride
with reflection padding. Letdk denote a Convolution-SpectralNorm-InstanceNorm-ReLU layer with filters and stride for down-sampling. Let uk be defined in the same manner as dk with transpose convolution for up-sampling. Let Rk denote a residual block of channel size across both layers. We use dilated convolution in the first layer of Rk with dilation factor of , followed by spectral normalization and instance normalization. The architecture of our generators is adopted from the model proposed by Johnson et al. :
The discriminators and follow the same architecture based on the PatchGAN [23, 57]. Let Ck-s denote a Convolution-SpectralNorm-LeakyReLU layer with filters of stride . The discriminators have the architecture C64-2, C128-2, C256-2, C512-1, C1-1. The final convolution layer produces scores predicting whether overlapping image patches are real or fake. LeakyReLU  is employed with slope .
We provide additional results produced by our model over the following datasets:
CelebA ( images)
Places2 ( million+ images)
Paris StreetView ( images)
With CelebA, we cropped the center of the images, then resized them to using bilinear interpolation. For Paris StreetView, since the images in the dataset are elongated (), we separate each image into three: 1) Left , 2) middle , 3) right , of the image. These images are scaled down to for our model, totaling images.
Table 4 shows the accuracy of our edge generator
across all three datasets. We measure precision and recall for various mask sizes. We emphasize that the goal of this experiment is not to achieve best precision and recall results, but instead to showcase how close the generated edges are to the oracle.
Tables 7 and 7 shows the performance of our model compared to existing methods over the datasets CelebA and Paris StreetView respectively. Our method produces noticeably better results. Note that we did not include values for PConv over the CelebA and Paris StreetView datasets as the source code was not available at the time of writing. Figures 13, and 14 display these results graphically. Additional inpainting results of our proposed model are shown in figures 15, 16, and 17.
Our source code, pre-trained models, and more results are available at:
In drawing, an edge is a boundary that separates two areas. A thick line brings the shape forward thin line indicates a plane receding into the background. In other words edges create a sense of distance and are not just about lines. Here we use HED  as an alternative edge detection system. Edges produced with HED, are of varying thickness, and pixels can have intensities ranging between 0 and 1. We noticed that it is possible to create edge maps that look eerily similar to human sketches by performing element-wise multiplication on Canny and HED edge maps. We compare the quantitative results between Canny and a combination of HED and Canny edges (i.e. HEDCanny). Generated images based on the combined edges gave the best performance. However, our generator is unable to generate these type of edges accurately during training. Table 5 shows trained on HEDCanny had the poorest performance out of all methods despite its ground truth counterpart achieving the best performance. These results suggest that better edge detectors result in better inpainting, however, effectively drawing those edges remains an open question in our research. Figure 12 shows the results of trained using hybrid edges.