Wavelet-Based Dual-Branch Network for Image Demoireing

07/14/2020 ∙ by Lin Liu, et al. ∙ 6

When smartphone cameras are used to take photos of digital screens, usually moire patterns result, severely degrading photo quality. In this paper, we design a wavelet-based dual-branch network (WDNet) with a spatial attention mechanism for image demoireing. Existing image restoration methods working in the RGB domain have difficulty in distinguishing moire patterns from true scene texture. Unlike these methods, our network removes moire patterns in the wavelet domain to separate the frequencies of moire patterns from the image content. The network combines dense convolution modules and dilated convolution modules supporting large receptive fields. Extensive experiments demonstrate the effectiveness of our method, and we further show that WDNet generalizes to removing moire artifacts on non-screen images. Although designed for image demoireing, WDNet has been applied to two other low-levelvision tasks, outperforming state-of-the-art image deraining and derain-drop methods on the Rain100h and Raindrop800 data sets, respectively.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 6

page 11

page 12

page 13

page 14

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

A smartphone has become an indispensable tool in daily life, and the popularity of mobile photography has grown supported by advancements in photo quality. It has become increasingly common to take photos of digital screens in order to quickly save information. However, when a photo is taken of a digital screen, moiré patterns often appear and contaminate the underlying clean image. These moiré patterns are caused by the interference between the camera’s color filter array (CFA) and the screen’s subpixel layout. In general, moiré patterns are likely to appear when two repetitive patterns interfere with each other. For example, moiré artifacts can appear when the repetitive patterns in textiles and building’s bricks interfere with camera’s CFA. Removing moiré patterns is challenging, as moiré patterns are irregular in shape and color, and can span a large range of frequencies. Unlike other image restoration tasks, such as image denoising [67, 27, 54], image demosaicing [9, 34]

and super resolution

[69, 22, 19, 20], where the challenge is mainly in removing high-frequency noise and recovering details, image demoiréing requires not only recovering high frequency image details, but also removing moiré patterns with frequencies spanning a large range.

Most existing image restoration methods [42, 21, 69] working in the RGB domain are not tailored to image demoiréing, though a few attempts [53, 32, 14] have been made to tackle it recently. Sun et al. [53] proposed a multi-resolution network to remove moiré patterns of different frequencies/scales. Liu et al. [32] developed a coarse-to-fine network and trained it on a synthetic dataset and refined real images with GAN. He et al. [14] utilized the edge information and appearance attributes of moiré patterns to demoiré. However, all the existing methods (working in the RGB domain) have difficulty in distinguishing moiré patterns from the real image content, and in dealing with low-frequency moiré patterns [53]. Fig. 1

shows some qualitative comparisons among four methods (including ours). We argue that image demoiréing can be more easily handled in the frequency domain. Wavelet-based methods have been explored in computer vision and shown good performance, e.g., in classification

[7, 38], network compression [28, 11], and face super-resolution [19]. However, due to their task-specific design, these methods cannot be directly used for image demoiréing.

Figure 1: Qualitative comparison among CFNet [32], U-Net [42], MultiscaleNet [53] and our network on the TIP2018 data set [53]. Ours is trained in the wavelet domain and removes the moiré patterns significantly better.

In this paper, we propose to remove moiré patterns in the frequency domain, where the input image with moiré patterns is first decomposed into different frequency bands using a wavelet transform. After the wavelet transform, moiré patterns are more apparent in certain wavelet subbands, where they can be more easily removed (see Fig. 3 for example). Our model, working in the frequency domain, is a dual-branch network with dense branches and dilated branches, which are responsible for restoring the close-range and far-range information, respectively. We also design a spatial attention mechanism called a direction perception module in the dense branches to highlight the areas with moiré patterns.

In this paper, we make following contributions:

  1. We propose a novel wavelet-based and dual-branch neural network for image demoiréing. We also propose a spatial attention mechanism called direction perception module (DPM) to highlight the areas with moiré patterns.

  2. Our network achieves the best results on the demoiréing task. Our trained model can also remove moiré artifacts on non-screen images.

  3. Our new architecture generalises well to other low-level vision tasks, such as deraining and deraindrop, where we also obtain state-of the-art results.

  4. In addition, we built a new urban-scene data set with more types of moiré patterns, which will be made publicly available.

2 Related Work

In this section, we give a brief review of the most relevant work for image demoiréing.

Moiré Pattern Removal. Early image demoiréing work [48, 43, 46, 33, 61] focused on certain specific moiré patterns. Sidorov et al. [48] presented a spectral model to address monotonous and monochrome moiré patterns. Other work tried to remove striped moiré artifacts [43, 56, 47] or dotted artifacts [46, 51, 25] in scanned or X-ray images. Yang et al. [61] and Liu et al. [33] removed moiré artifacts using low-rank and sparse matrix decomposition. But their methods focus on textile moiré patterns and cannot handle low-frequency moiré patterns. Compared to these methods, our approach can deal with a much larger range of moiré patterns as we do not make presumptions about the moiré patterns. Contemporary models [53, 32, 14, 70] and recent challenges [65, 66, 64] cast image demoiréing as an image restoration problem addressed using deep learning. Recently Liu et al. [32]

built a coarse-to-fine convolutional neural network to remove moiré patterns from photos taken of screens. But the method mainly used synthetic data for network training and focuses on removing moiré patterns in text scenes. Sun

et al. [53] proposed a multi-resolution convolutional neural network for demoiréing and released an associated dataset. He et al. [14] labeled the data set in [53] with three attribute labels of moiré patterns, which is beneficial to learn diverse patterns. This method, along with [32, 53], works in the RGB domain and in a multi-scale manner. Such a strategy has limited capacity to correctly distinguish moiré patterns from true image content. In contrast, our method works in the wavelet domain, and we introduce a novel network design, resulting in stronger moiré pattern removal while restoring image details.

Wavelet-based Methods. Wavelet-based methods have been explored in some computer vision tasks, including classification [7, 38, 29, 57], network compression [28, 11], face aging [36], super-resolution [19, 35], style transfer [63], etc. Fujieda et al. [7]

proposed wavelet CNNs that utilize spectral information to classify the textures. Liu

et al. [36]

used a wavelet-based method to capture age-related texture details at multiple scales in the frequency domain. Our work is the first to use a wavelet transform to remove moiré patterns in the frequency domain. Compared with Fourier transform and discrete cosine transform, wavelet transform considers both spatial domain information and frequency domain information. With the wavelet composition, different wavelet bands represent such a broad range of frequencies, which can not be achieved by a few convolutional layers even with large kernels.

The most relevant in the above studies is [19] for using wavelets for face super-resolution, where a neural network is deployed to predict the wavelet coefficients. But predicting the wavelet coefficients of a moiré-free image from its moiré image in the RGB domain is difficult. Moiré patterns cover a wide range in both space and frequency domains, making it hard to distinguish moiré patterns from true scene textures. For photo-realistic style transfer, Yoo et al. [63] regarded the wavelet transform and wavelet inverse transform as a pooling layer and an unpooling layer respectively, with the aim of preserving their structural information. However, this technique [63] may not be suitable for image demoiréing, where moiré patterns can have strong structural information overlaid with true image contents. Our work runs directly in the wavelet domain. Wavelet transform and inverse wavelet transform are used outside of the network.

Dual-branch Design. Dual-branch network structure design makes use of two branches, complementing each other [8, 40, 6, 45]. It has been used in image super-resolution [40], classification [8], segmentation [6] and person-re-ID [45]. Gao et al. [8] proposed a dual-branch network for polarimetric synthetic aperture radar image classification. One branch is used to extract the polarization features and the other to extract the spatial features. Fu et al. [6] proposed a dual attention network with a position attention module and a channel attention module for scene segmentation. For image super-resolution, the dual branch in [40] can give consideration to easy image regions and hard image regions at the same time. All existing dual-branch networks have different branches, each focusing on certain image features, which only merge at the end of the network. In our design, the two branches iteratively communicate with each other to achieve a better representation.

Texture Removal. Texture removal [50, 13, 68] is related to demoiréing, as moiré patterns can be viewed as a special type of texture. Methods using local filters attempted to remove certain textures while maintaining other high-frequency structures [50, 13, 68]. Subr et al. [50] smoothed texture by averaging two manifolds from local minima and maxima. Both Ham et al. [13] and Zhang et al. [68] used dynamic guidance and a fidelity term for image filtering. Multiple strategies to smooth texture were used in [49, 24, 1]. Optimization-based methods were also exploited [52, 59, 58]. Sun et al. [52] and Xu et al. [58] used minimization to retrieve structures from images. Xu et al. [59] optimized a global function with relative total variation regularization. However, these texture removal methods focus on the structure of the image and remove some repeated and identical or similar patterns. They do not address moiré patterns that span a much wider range of frequencies and appear in varied shapes and directions.

3 Our Method

The architecture of WDNet is shown in Fig. 2. The original RGB image input is first transformed into the Wavelet domain, where a Dual-branch Network is used to remove the moiré patterns. The final RGB image output is obtained by applying the inverse wavelet transform to the network’s output. The dual-branch network has seven dual-branch modules, each including a dense branch and a dilation branch. A ResNet [15] style skip connection is made. The dense and dilation branches are responsible for acquiring moiré information in different frequency ranges; the former detects close-range patterns and the latter detects far-range patterns.

Figure 2: The architecture of the proposed WDNet (a) consisting of two kinds of branches, the dense branch (b) for restoring the close-range information and the dilation branch (c) for restoring the far-range information. Working in the wavelet domain can better remove the moiré patterns and retain the details. The numbers of channels are given in the figure.

3.1 Working in the Wavelet Domain

Our network operates in the wavelet domain to remove moiré patterns. We employ 2D fast wavelet transform (FWT) to decompose the input RGB image into a sequence of wavelet subbands (coefficients) of a common smaller size, but with different frequency content. We choose the simplest Haar wavelet as the basis for the wavelet transform. A Haar wavelet transform can be efficiently computed and suitable for our task. The FWT iteratively applies low-pass and high-pass decomposition filters along with downsampling to compute the wavelet coefficients where the low-pass filter = and the high-pass filter = . In each level of the transform, we use the high-pass filter and the low-pass filter along the rows to transform an image to two images, and then we apply the same filters along the columns of these two images, obtaining four images in total. The equations to derive the subbands can be found in [10].

As shown in Figs. 3(c) and (d), the wavelet subbands at different levels correspond to different frequencies. Moiré patterns are conspicuous only in certain wavelet subbands. For example, in Fig. 3(e), the difference between (c) and (d) shows that the first three wavelet subbands of the first column in Fig. 3(c) contain obvious and different moiré patterns. From our experiments, we find that wavelet subbands with higher frequencies usually contain fewer moiré patterns. These wavelet subbands provide important information to recover the details for moiré pattern-free images.

In our method, the wavelet transform level is set to 2. The original RGB input is transformed into 48 subbands, with 16 subbands for each color channel. As such, another benefit of working in the wavelet domain is that the spatial size of the original image is reduced by a quarter in both the width and height . The reduced spatial size consequently reduces the computation in the deep network. Besides, we concatenate all the 48 channels instead of processing the high and low frequency bands individually, because moire patterns for each image vary a lot, it is difficult to find a threshold to manually separate them.

Figure 3: (a) An image with moiré patterns. (b) The ground truth of (a). (c) Wavelet subbands transformed from the gray-level image of (a). (d) Wavelet subbands transformed from the gray-level image of (b). (e) Difference between (c) and (d).

3.2 Dense Branch

DenseNet [18] can alleviate the problem of vanishing gradient and reduce the number of network parameters through the design of bypass connections and feature reuse. In low-level vision, dense blocks have been used in deraining and dehazing [44, 12], and image super-resolution with the Residual Dense Network (RDN) [69] or other networks [22].

As shown in Fig. 2(b), we design each dense branch by adding a new Direction Perception Module (DPM) to the residual dense module borrowed from RDN [69]. The residual dense module has two small dense blocks each with 5 convolutional layers. In the same dense block, the input of each layer contains the output of all previous layers. DPM is used to find the directions of moiré patterns and will be explained in Section 3.4 in detail. The output of DPM and each dense block are multiplied, weighted by a factor , and then the result is added to the input. This design can effectively find the locations of close-range moiré patterns.

3.3 Dilation Branch

Subsampling or pooling feature maps can enlarge receptive fields, but loses some details. Dilated convolution is used to overcome this problem. In each of our dilation branches (Fig. 2(c)), there are two layers: a dilated convolution layer with a dilation rate , followed by a traditional convolution, where is the index of a dilation branch with and . The rationale for this design is explained below.

When multiple dilated convolutions with a fixed dilation rate (say, 2) are applied successively, many pixels are not involved in the convolutions and a special kind of artifact called gridding appears [55]. Alternatively, a hybrid design using dilation rates (1,2,3,1,2,3,…) [55] is not suitable for our task because these dilations are not large enough to cover general moiré patterns. Therefore, we design our dilation rates according to the Fibonacci series (e.g., (1,2,3,5,8,13,21) when there are 7 dilation branches). Additionally, we apply a traditional convolution on the output of the dilated convolution to further reduce any gridding artifact (Fig. 2(c)).

Figure 4:

(a) Image recurrent neural network (IRNN). (b) Our direction perception module (DPM) with two stages, each with 8 directions.

3.4 Direction Perception Module

We propose a Direction Perception Module (DPM) by improving the Image Recurrent Neural Network (IRNN) [2, 60, 17], as shown in Fig. 4. The two-stage four-directional IRNN (Fig. 4(a)) architecture enhances the use of contextual information, where the first stage in IRNN aims to produce feature maps that acquire neighboring contextual information and the second stage in IRNN further gathers the global information. However, in IRNN, in the feature maps at the end of the first stage, a pixel can only obtain information in the vertical and horizontal directions, but not in other directions.

As shown in Fig. 4(b), we improve IRNN in two aspects. First, we extend the 4-direction perception to 8-direction perception by including 4 more diagonal directions to enhance the detection ability of slanting moiré patterns. The moiré patterns in other directions are detected in the second stage. Second, the weight maps are used to distinguish the importance of information in different directions. Unlike IRNN which learns direction-aware features in the embedded space, we use DPM to generate the attention map which highlights moiré pattern spatial distributions and is supervised by discussed in Sec. 3.6. The combination of the two convolution layers and the eight-direction perception blocks is equivalent to an attention operation.

3.4.1 Loss Function.

The loss function of the whole network is defined as follows:

(1)

where is the loss between the image and its ground-truth in the RGB domain, is the perceptual loss [23], and the other parts are described below.

3.4.2 Wavelet Loss.

Let and

be the ground-truth and the estimated wavelet subbands, respectively. Two wavelet-based losses, MSE loss

and detail loss , are defined as follows:

(2)

where the first term is for the low-frequency components, and the second for the high-frequency ones, and and are two weighting factors.

(3)

which is inspired by the work in super-resolution [19] to prevent the wavelet coefficients from converging to 0 and therefore retaining the high-frequency details of the image. is a positive number and set to 1.2 in this work. Setting to a value greater than 1 can effectively prevent from converging to 0. Different from the loss function in [19], ours does not have a bias value , because we found that has little effect on the final results. Besides, the value of in [19] was set to 0 in their source code. The network’s wavelet loss is then:

(4)

3.4.3 Attention Loss.

We further apply the attention loss as:

(5)

where is the output of the DPM in the network and is the mask of the moiré patterns, which is computed by thresholding the difference between the moiré-contaminated image and the ground-truth. is a downsampling operation to ensure that the mask and have the same size. The threshold is set to 15 in this work.

4 Experiments

We compare WDNet with state-of-the-art methods, and conduct user studies for image demoiréing. We also demonstrate that WDNet when trained only for screen image demoiréing can also remove moiré artifacts on non-screen images. Finally, we apply WDNet to two other tasks, deraining and de-raindrop, to show that our method generalizes well to other frequency-based low-level vision tasks.

4.1 Implementation Details

Throughout our experiments, the input is a RGB image, which is then transformed into 48 different-level wavelet subbands with the size of

by a Haar wavelet transform. We use seven dual-branch modules and add DPM into the 4th dense branch to perceive moiré patterns. Our model is implemented in PyTorch and runs on a NVIDIA Tesla V100 GPU. The batch size is set to 16 during training. Adam

[26]

is used to search the minimum of the loss function. The learning rate of the generator is 0.0002 and reduced by one-tenth for every 20 epochs. In Eqns. (

1) and (2), the parameters , , , , and are empirically set to 1, 5, 10, 200, 0.01 and 1.1 respectively. The method of eye initialization in [69] is used to initialize the weights of WDNet. The value of throughout the paper is set to 0.2. The dense branch has more parameters than the dilation branch. Setting to 0.2 makes the weights of the dilation branch’s parameters larger, which can maintain the details of the original image.

4.2 Datasets and State-of-the-Art Methods

We perform experiments on two datasets, TIP2018 [53] and a new London’s Buildings. The TIP2018 dataset contains 100,000+ image pairs with about

resolution and the moiré-free images come from the ImageNet ISVRC 2012 dataset. The new dataset London’s Buildings was built by us recently and will be publicly released. It is an urban-scene data set and its images contain bricks, windows and other regular patterns which are prone to generate moiré patterns. We use mobile phones and screens different from those used in the TIP2018 data set and thus provide additional diversity of moiré patterns. London’s Buildings includes 400 training pairs and 60 testing pairs with about

resolution. The images used in training are resized (in TIP2018) or cropped (in London’s buildings) from the original higher-resolution images. We conduct two experiments on the TIP2018 data set, TestA and TestB. TestB follows the same setting as in [53]. TestA is a subset of TestB and this subset contains 130 hard samples from the same camera.

We compare our model with MuiltiscaleNet [53], CFNet [32]

, Pix2pix 

[21], U-Net [42], MopNet [14], and ResNet-34 [15]. For fair comparison, when a network is trained on the TIP2018 data set, we use the same setting as in [53]. We faithfully reimplement MuiltiscaleNet [53] and CFNet [32] based on their papers. Note that since MopNet is very recent work without its code released, we can only compare with it in one experiment where [14] gives its result on TIP2018.

4.3 Comparison with the State-of-the-Art

For quantitative comparison, we choose the PSNR and SSIM. As shown in Tables 1 and 2, our method outperforms the state-of-the-art on both datasets. Note that our model outperforms the most recent MopNet [14] by 0.33dB, even though MopNet uses additional annotation (labelling TIP2018 with three attribute labels of moiré patterns) for training. We show some qualitative results in Figs. 5 and 6, where there are different kinds of moiré patterns appearing in the input images. Our WDNet most effectively removes the moiré patterns. For the computation consumption, CFNet, MultiscaleNet, U-Net, MopNet and WDNet take 58.2ms, 5.7ms, 3.8ms, 71.8ms and 14.6ms, respectively, to restore a image on one NVIDIA Tesla P100 GPU.

Test set Pix2pix U-Net CFNet MultiscaleNet MopNet WDNet
TestA PSNR/SSIM 25.32/0.756 25.80/0.803 25.52/0.810 26.11/0.801 –/– /
TestB PSNR/SSIM 25.74/0.825 26.49/0.864 26.09/0.863 26.99/0.871 27.75/0.895 /

Table 1: Quantitative comparison on TIP2018. The best results are in bold.
U-Net MultiscaleNet ResNet CFNet WDNet
PSNR/SSIM 23.48/0.790 23.64/0.791 23.87/0.780 23.22/0.764
Table 2: Quantitative comparison on London’s Building.
WDNetCFNet WDNetU-Net WDNetMultiscaleNet
Preference rate 83.7% 82.1% 80.5%
Table 3: User study. The preference rate shows the percentage of comparisons in which the users prefer our results.
Figure 5: Visual comparison among Pix2pix [21], CFNet [32], U-Net [42], MuiltiscaleNet [53] and our model WDNet, evaluated on images from the TIP2018 data set. The numbers under the images are PSNR/SSIM.
Figure 6: Visual comparison on an image from the London’s Building data set.
Figure 7: Some results on non-screen images.
Figure 8: The heat maps of the feature maps outputted from the two branches of the 6th layer.
Figure 9: Visual comparison between WDNet and WDNet without wavelet transform. These patches show that the former restores the details better and removes more moiré patterns.

4.3.1 User Studies

We also conduct a user study with 10 users to compare the visual quality of the results generated by the models. During the study, two images and , respectively obtained by our model and another model, and the corresponding moiré-free image (ground truth) , are shown to a user who is asked to choose one from and which is closer to . There are 100 triplets of these images randomly selected from TIP2018 to compare two models and the results are reported in Table 3.

4.3.2 Removal of Moiré Artifacts on Non-screen Images

Moiré artifacts may also appear on objects with dense repetitive textures, such as buildings and clothes. This kind of moiré artifacts usually does not always appear on the whole image. In Fig. 7, we show some results of directly applying our WDNet model trained on TIP2018 without fine-tuning on non-screen images. It shows that our method is effective in removing this kind of moiré patterns as well.

4.4 Ablation Studies

To conduct ablation studies to quantify the contributions of the components of WDNet, the TestA dataset is used in the experiments. To understand the restoration strategy between the two branches in the dual-branch architecture, we examine the feature maps of WDNet. At layer six, we normalize each channel of the feature map, and then sum up the values channel-wise to obtain a heat map. As shown in Fig. 8, the dense branch focuses on the local moiré patterns (interlaced bright and dark areas), while the dilation branch focuses on the whole areas contaminated by the moiré patterns due to its larger receptive field. It demonstrates that the two types of branches are responsible for the acquisition of the close-range and far-range information and jointly restore the image.

To test the effectiveness of the dense branches, we replace the dense blocks of them with ResNet blocks. To test the dilation branches, we replace the dilated convolutions with normal convolutions. As shown in Table 4, both the dense and dilation branches play an important role in WDNet. When the dense branches or the dilation branches are replaced, the PSNR decreases by 0.52dB or 1.05dB, respectively. The PSNR decreases by 0.41dB when we remove the DPM module and the attention loss, and by 0.27dB when we remove the weight maps in DPM. In addition, to compare the 8-directional DPM with the 4-directional IRNN, using the latter makes the PSNR decrease by 0.19dB.

In order to verify whether working in the wavelet domain is better than in the RGB domain, we design a network by removing the wavelet transform and wavelet inverse transform from WDNet, without the wavelet loss during training. These two networks are compared at the same computation amount and the results are shown in Table 4. A visual comparison is shown in Fig. 9. From Table 4 and Fig. 9, we can see that the complete model best achieves moiré pattern removal. The wavelet transform can help to keep the edges and prevent the details from blurring.

The above studies clearly demonstrate that all the WDNet’s components and the network design have their contributions to the excellent performance of WDNet. We also perform an experiment to explore the effect of different wavelet transform levels on the network’s performance with the same network parameters. For levels 1, 2 and 3, the PSNR/SSIM are 27.24/0.866, 27.88/0.863, and 27.94/0.843, respectively. As the number of levels increases, the PSNR increases, but the SSIM decreases. The reason for such an interesting phenomenon will be left as the future work. Some recent related work such as [3] shows that perceptual quality and distortion (PSNR) may conflict with each other, particularly for image restoration tasks.

Figure 10: Frequency spectra of images of rain streak, rain drop and moiré pattern.
Network a b c d e Complete model
PSNR/SSIM 26.83/0.858 27.36/0.856 27.03/0.859 27.47/0.853 27.69/0.857 27.88/0.863
Table 4: Ablation study on TestA. (a) Replace dilated convolutions with normal convolutions. (b) Replace the dense blocks with the residual blocks. (c) Without the wavelet and inverse wavelet transfrom. (d) Without DPM. (e) Replace the 8-directional DPM with the 4-directional IRNN.

4.5 Extension to Deraining and Deraindrop

We show that our network also has advantages over the state-of-the-art methods in some other low-level vision tasks, such as deraining and deraindrop. Deraining methods can be divided into traditional methods and deep-learning methods. Traditional methods [37, 4] remove rain streaks in images by designing some hand-crafted priors. Deep-learning methods [62, 30, 16] use convolutional neural networks to learn the rain streak features automatically. Yang et al. [62] created a multi-task network to jointly detect and remove rain. And Li et al. [30] proposed a recurrent neural network, where useful information in previous stages is beneficial to the rain removal in later stages. However, few deep-learning methods study how to use the feature of the rain streaks in the frequency domain to derain. In Fig. 10, similar to moiré patterns, rain streaks include high-frequency and low-frequency patterns, depending on the severity of the rain and distance to the camera. However, unlike moiré patterns, which can have different orientations in the same image, the directions of rain streaks are more consistent, resulting in an easier image restoration task. Like demoiréing, after the wavelet transform, different types of rain steaks in the RGB domain are easier to distinguish in the wavelet domain.

We train WDNet on the Rain100h data set [62] for deraining and on the Raindrop800 data set [39] for deraindrop, and test it on the corresponding test sets. Though the textures of the rain/raindrop and moiré patterns are very different in color and structure, our WDNet method can also remove them well. The results are given in Table 5, Table 6 and Fig. 11, which show that our WDNet performs better than the state-of-the-art. More visual results are given in the supplementary material.

Method DSC [37] JORDER_R [62] DID_MDN [69] RESCAN [30] DAFNet [16] WDNet(Ours)
PSNR/SSIM 15.6/0.5446 23.45/0.749 24.53/0.799 26.45/0.845 28.44/0.874 /
Table 5: Quantitative comparison on Rain100h.
Method Eigen [5] Pix2pix [21]  [31] AttentiveGAN [39] Quan’s [41] WDNet (Ours)
PSNR/SSIM 23.90/0.80 28.85/0.84 30.79/0.93 31.52/0.92 31.44/ /
Table 6: Quantitative comparison on Raindrop800.
Figure 11: (a) Visual deraining comparison among JORDER_R, RESCAN and WDNet. (b) Visual deraindrop comparison between AttentiveGAN and WDNet.

5 Conclusion

We have proposed a novel wavelet-based dual-branch network (WDNet) to remove moiré patterns. Working in the wavelet domain can restore more details and is more effective at moiré pattern removal than in the RGB domain. WDNet’s dense branches and dilation branches are responsible for the acquisition of close-range and far range information, respectively. The new DPM module inside some dense branches can better perceive slanting moiré patterns. WDNet substantially outperforms other state-of-the-art models. Moreover, it obtains the best results in two other low-level vision tasks, deraining and deraindrop. In addition, we built a new urban-scene data set with challenging types of moiré patterns, which will be made publicly available. Our future work includes applications of WDNet on other vision tasks such as denoising and demosaicing.

References

  • [1] Bao, L., Song, Y., Yang, Q., Yuan, H., Wang, G.: Tree filtering: Efficient structure-preserving smoothing with a minimum spanning tree. TIP, 2013
  • [2] Bell, S., Lawrence Zitnick, C., Bala, K., Girshick, R.: Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In: CVPR, 2016
  • [3] Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: CVPR, 2018
  • [4] Chang, Y., Yan, L., Zhong, S.: Transformed low-rank model for line pattern noise removal. In: ICCV, 2017
  • [5] Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In: CVPR, 2013
  • [6] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: CVPR, 2019
  • [7] Fujieda, S., Takayama, K., Hachisuka, T.: Wavelet convolutional neural networks for texture classification. arXiv preprint arXiv:1707.07394, 2017
  • [8] Gao, F., Huang, T., Wang, J., Sun, J., Hussain, A., Yang, E.: Dual-branch deep convolution neural network for polarimetric sar image classification. Applied Sciences, 2017
  • [9] Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. TOG, 2016
  • [10] Gonzalez R. C., W.R.E.: Digital image processing (3rd edition). In: Prentice-Hall, Inc. 2007
  • [11] Gueguen, L., Sergeev, A., Kadlec, B., Liu, R., Yosinski, J.: Faster neural networks straight from jpeg. In: NeurIPS, 2018
  • [12] Guo, T., Cherukuri, V., Monga, V.: Dense123’color enhancement dehazing network. In: CVPRW, 2019
  • [13] Ham, B., Cho, M., Ponce, J.: Robust image filtering using joint static and dynamic guidance. In: CVPR, 2015
  • [14] He, B., Wang, C., Shi, B., Duan, L.Y.: Mop moire patterns using mopnet. In: ICCV, 2019
  • [15] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, 2016
  • [16] Hu, X., Fu, C.W., Zhu, L., Heng, P.A.: Depth-attentional features for single-image rain removal. In: CVPR, 2019
  • [17] Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. TPAMI,2019
  • [18] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR, 2017
  • [19] Huang, H., He, R., Sun, Z., Tan, T.: Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In: ICCV, 2017
  • [20] Isobe, T., Li, S., Jia, X., Yuan, S., Slabaugh, G., Xu, C., Li, Y.L., Wang, S., Tian, Q.: Video super-resolution with temporal group attention. In: CVPR, 2020
  • [21]

    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, 2017

  • [22] Jang, D.W., Park, R.H.: Densenet with deep residual channel-attention blocks for single image super resolution. In: CVPRW, 2019
  • [23] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV, 2016
  • [24] Karacan, L., Erdem, E., Erdem, A.: Structure-preserving image smoothing via region covariances. TOG, 2013
  • [25] Kim, T.H., Park, S.I.: Deep context-aware descreening and rescreening of halftone images. TOG, 2018
  • [26] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
  • [27] Lefkimmiatis, S.: Universal denoising networks: a novel cnn architecture for image denoising. In: CVPR, 2018
  • [28]

    Levinskis, A.: Convolutional neural network feature reduction using wavelet transform. Elektronika ir Elektrotechnika, 2013

  • [29] Li, Q., Shen, L., Guo, S., Lai, Z.: Wavelet integrated cnns for noise-robust image classification. In: CVPR, 2020
  • [30] Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, 2018
  • [31] Lin, H., Fu, X., Jing, C., Ding, X., Huang, Y.: A2̂ net: Adjacent aggregation networks for image raindrop removal. arXiv preprint arXiv:1811.09780, 2018
  • [32] Liu, B., Shu, X., Wu, X.: Demoiréing of camera-captured screen images using deep convolutional neural network. arXiv preprint arXiv:1804.03809, 2018
  • [33] Liu, F., Yang, J., Yue, H.: Moiré pattern removal from texture images via low-rank and sparse matrix decomposition. In: VCIP, 2015
  • [34] Liu, L., Jia, X., Liu, J., Tian, Q.: Joint demosaicing and denoising with self guidance. In: CVPR, 2020
  • [35] Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-cnn for image restoration. In: CVPRW, 2018
  • [36] Liu, Y., Li, Q., Sun, Z.: Attribute-aware face aging with wavelet-based generative adversarial networks. In: CVPR, 2019
  • [37] Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: ICCV, 2015
  • [38] Oyallon, E., Belilovsky, E., Zagoruyko, S.: Scaling the scattering transform: Deep hybrid networks. In: ICCV, 2017
  • [39] Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: CVPR, 2018
  • [40] Qin, J., Xie, Z., Shi, Y., Wen, W.: Difficulty-aware image super resolution via deep adaptive dual-network. In: ICME, 2019
  • [41] Quan, Y., Deng, S., Chen, Y., Ji, H.: Deep learning for seeing through window with raindrops. In: ICCV, 2019
  • [42] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI, 2015
  • [43] Sasada, R., Yamada, M., Hara, S., Takeo, H., Shimura, K.: Stationary grid pattern removal using 2d technique for moiré-free radiographic image display. In: Medical Imaging, 2003
  • [44] Shen, L., Yue, Z., Chen, Q., Feng, F., Ma, J.: Deep joint rain and haze removal from a single image. In: ICPR, 2018
  • [45] Si, J., Zhang, H., Li, C.G., Kuen, J., Kong, X., Kot, A.C., Wang, G.: Dual attention matching network for context-aware feature sequence based person re-identification. In: CVPR, 2018
  • [46] Siddiqui, H., Boutin, M., Bouman, C.A.: Hardware-friendly descreening. TIP, 2009
  • [47] Sidorov, D.N., Kokaram, A.C.: Removing moire from degraded video archives. In: ESPC, 2002
  • [48] Sidorov, D.N., Kokaram, A.C.: Suppression of moiré patterns via spectral analysis. In: VCIP, 2002
  • [49] Su, Z., Luo, X., Deng, Z., Liang, Y., Ji, Z.: Edge-preserving texture suppression filter based on joint filtering schemes. TMM, 2012
  • [50] Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. In: TOG, 2009
  • [51] Sun, B., Li, S., Sun, J.: Scanned image descreening with image redundancy and adaptive filtering. TIP, 2014
  • [52] Sun, Y., Schaefer, S., Wang, W.: Image structure retrieval via minimization. TVCG, 2017
  • [53] Sun, Y., Yu, Y., Wang, W.: Moiré photo restoration using multiresolution convolutional neural networks. TIP, 2018
  • [54] Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: ICCV, 2017
  • [55] Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., Cottrell, G.: Understanding convolution for semantic segmentation. In: WACV, 2018
  • [56] Wei, Z., Wang, J., Nichol, H., Wiebe, S., Chapman, D.: A median-gaussian filtering framework for moiré pattern noise removal from x-ray microscopy image. Micron, 2012
  • [57] Williams, T., Li, R.: Wavelet pooling for convolutional neural networks. In: ICLR, 2018
  • [58] Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via l 0 gradient minimization. In: TOG, 2011
  • [59] Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. TOG, 2012
  • [60] Xu, X., Sun, D., Liu, S., Ren, W., Zhang, Y.J., Yang, M.H., Sun, J.: Rendering portraitures from monocular camera and beyond. In: ECCV, 2018
  • [61] Yang, J., Liu, F., Yue, H., Fu, X., Hou, C., Wu, F.: Textured image demoiréing via signal decomposition and guided filtering. TIP, 2017
  • [62] Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: CVPR, 2017
  • [63] Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.W.: Photorealistic style transfer via wavelet transforms. In: CVPR, 2019
  • [64] Yuan, S., Timofte, R., Leonardis, A., Slabaugh, G.: Ntire 2020 challenge on image demoireing: Methods and results. In: CVPRW, 2020
  • [65] Yuan, S., Timofte, R., Slabaugh, G., Leonardis, A.: Aim 2019 challenge on image demoireing: Dataset and study. In: ICCVW, 2019
  • [66] Yuan, S., Timofte, R., Slabaugh, G., Leonardis, A., et al.: Aim 2019 challenge on image demoireing: Methods and results. In: ICCVW, 2019
  • [67] Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017
  • [68] Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: ECCV, 2014
  • [69] Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR, 2018
  • [70] Zheng, B., Yuan, S., Slabaugh, G., Leonardis, A.: Image demoireing with learnable bandpass filters. In: CVPR, 2020