Structural Residual Learning for Single Image Rain Removal

by   Hong Wang, et al.
Xi'an Jiaotong University

To alleviate the adverse effect of rain streaks in image processing tasks, CNN-based single image rain removal methods have been recently proposed. However, the performance of these deep learning methods largely relies on the covering range of rain shapes contained in the pre-collected training rainy-clean image pairs. This makes them easily trapped into the overfitting-to-the-training-samples issue and cannot finely generalize to practical rainy images with complex and diverse rain streaks. Against this generalization issue, this study proposes a new network architecture by enforcing the output residual of the network possess intrinsic rain structures. Such a structural residual setting guarantees the rain layer extracted by the network finely comply with the prior knowledge of general rain streaks, and thus regulates sound rain shapes capable of being well extracted from rainy images in both training and predicting stages. Such a general regularization function naturally leads to both its better training accuracy and testing generalization capability even for those non-seen rain configurations. Such superiority is comprehensively substantiated by experiments implemented on synthetic and real datasets both visually and quantitatively as compared with current state-of-the-art methods.


page 5

page 6

page 7

page 8

page 9

page 10

page 11

page 12


Semi-supervised CNN for Single Image Rain Removal

Single image rain removal is a typical inverse problem in computer visio...

A Survey on Rain Removal from Video and Single Image

Rain streaks might severely degenerate the performance of video/image pr...

From Rain Removal to Rain Generation

Single image deraining is an important yet challenging issue due to the ...

A Model-driven Deep Neural Network for Single Image Rain Removal

Deep learning (DL) methods have achieved state-of-the-art performance in...

Deep Joint Rain Detection and Removal from a Single Image

In this paper, we address a rain removal problem from a single image, ev...

A Coarse-to-Fine Multi-stream Hybrid Deraining Network for Single Image Deraining

Single image deraining task is still a very challenging task due to its ...

RCDNet: An Interpretable Rain Convolutional Dictionary Network for Single Image Deraining

As a common weather, rain streaks adversely degrade the image quality. H...

I Introduction

Images captured in rainy days often suffer from noticeable degradation of scene visibility and adversely affect the performance of subsequent image processing tasks, such as video surveillance [36, 1] and intelligent vehicles [37]. Removing rain streaks from a rainy image thus has become a necessary step for a wide range of practical applications, and has attracted much research attention recently [26, 47].

Fig. 1: Illustration of the proposed structural residual network (SRNet), which consists of three parallel subnetworks with similar structures but different dilated factors (DF) to capture rational structures expressing rain streaks.

Many single image deraining methods have been raised in the recent decades. Some early attempts exploited different filters to decompose a rainy image into low frequency part (LFP) and high frequency part (HFP), such as guided filter [20, 12] and smoothing filter [4]

, and then restored the rain-free image by combining the LFP and texture details. Recently, many researchers further explored the physical properties of rain and background layers and formulated them into different prior terms for single image rain removal. For example, the techniques of Gaussian mixture model (GMM) 

[28] and discriminative sparse coding (DSC) [49] have been used to model rain layers. Besides, joint convolutional analysis and synthesis sparse representation (JCAS) [11] is a typical instance that describes rain and background layers with concise mathematical models. These methods have been shown to be effective in certain scenarios. Such prior-model-based techniques, however, are still lack of the ability to flexibly adapt to rainy images with complicated rain shapes and background scenes. Besides, these methods are generally time-consuming due to their inevitable iterative optimization computations, which is always unfriendly to real users.

Very recently, driven by the success of deep learning (DL) techniques in low level vision tasks, the related techniques have also been employed in the deraining task. They are routinely required to pre-collect abundant training samples with rainy-clean image pairs to learn non-linear mapping from rainy image to its rain-free one in an end-to-end manner. The most representative methods along this research line include DerainNet [6], deep detail network (DDN) [7], deep joint rain detection and removal network (JORDER_E) [45, 46], recurrent squeeze-and-excitation context aggregation network (RESCAN) [27], progressive image deraining network (PReNet) [34], and spatial attentive network (SPANet) [41].

Albeit achieving success in certain contexts, the current DL methods for single image deraining still exist evident limitations. Specifically, the effectiveness of these methods largely relies on the quality and quantity of pre-collected training samples which are composed of large amount of rainy-clean images simulating the network inputs and outputs. The training images should include possibly wide range of rain shapes so as to cover those potentially occurring in testing stages. On the one hand, the non-linear mapping represented by the deep network architecture brings strong fitting capability on approximating abundant rain structures contained in training images beyond conventional model-based approaches. On the other hand, however, their complex network expression also tends to conduct redundancy in expressing rain streaks in training samples and excessive flexibility on adapting to newly input rainy images. A DL model achieving good training accuracy might not finely extract rains (like those containing evident background scenes, as shown in our experiments) from some practical rainy images with complex rain streaks non-seen in training stages. How to alleviate such generalization issue has become the core issue nowadays in current single image deraining research [41].

Against this issue, we build a specific network architecture, called structural residual network (SRNet), by enforcing its output residual finely accordant with the prior expressions for general rains. Such a structural residual setting forms a strong regularizer for rain layer extraction, guaranteeing the rationality of output rain streaks even for non-seen rain shapes differentiated from those contained in training samples, and is thus expected to achieve generalization ability in testing stages.

Specifically, inspired by the previous research on rain structure modeling, we summarize the following prior knowledge for representing general rain streaks, and employ them for building our network architecture. Firstly, as explored in [11] and  [14], rain streaks always repeatedly appear at different locations over a rainy image with similar local patterns like shape, thickness, and direction. Such knowledge can be delivered by convolutional representation under a set of local filters across the entire images. The structures of these filters are expected to capture local repetitive patterns inside rains. Secondly, an evident prior of rains is that they are always sparsely scattered over an image [49, 57]. This knowledge can be expressed by the sparsity of feature maps (can be seen as coefficients) convoluted with those local filters. In our network, each feature map of the last layer is yielded by an embedded encoder-decoder subnetwork, in which the joint MaxPooling and MaxUnpooling operation naturally leads to the sparsity of all last-layer feature maps. Thirdly, another common prior used for describing rain streaks are their multi-scale characteristics [25], referring to the fact that rain streaks in different distances appear with different sizes in a rainy image. We thus construct the output of our network as an integration of multiple encoder-decoder subnetworks through utilizing dilated convolutions [48] with different dilated factors. This facilitates a rational expression for extracted rain layers with different scales. The architecture is demonstrated in Fig. 1 for easy observation.

Note that such a structural residual setting keeps the rain layer extracted by the network comply with the prior knowledge underlying general rains, and thus is functioned like a conventional regularizer to rectify output rain shapes being well extracted from rainy images by the network in both training and predicting stages. This naturally leads to both its potential good training accuracy and testing generalization capability even for those non-seen rain configurations. Such superiority of the proposed network is substantiated by comprehensive experiments implemented on synthetic and real datasets, especially on its fine generalization capability. Furthermore, ablation studies have been provided to verify the necessity of all modules involved in our network.

The paper is organized as follows. Section II briefly reviews related works. Section III presents the proposed structural residual single image deraining network as well as the network training details. Comprehensive experiments are shown in Section IV and the paper is finally concluded.

Ii Related Work

In this section, we briefly review current development along video deraining methods and single image deraining methods.

Ii-a Video Deraining Methods

Garg and Nayar [9, 10] initially discussed the visual effects of raindrops on imaging systems and proposed a video deraining method, by capturing dynamics of raindrops through a space-time correlation model and describing the photometry of rain via a motion blur model. Later, researchers made more investigations on exploring physical properties of rain streaks for video deraining, like temporal-chromatic [54, 32] and spatio-temporal frequency characteristics [2, 40].

In the past years, more intrinsic prior structures of rain streaks and background scenes of rainy videos have been analyzed and formulated for deraining algorithm design. For example, Chen et al. [3] investigated the non-local similarity and repeatability of rain streaks and provided a low rank model. To remove rain and snow from a video, Kim et al. [19] adopted a low rank matrix completion strategy. To handle heavy rain streaks and dynamic scenes, Ren et al. [35] decomposed rain streaks into two classes: sparse ones and dense ones, and detected them with different models. Recently, Wei et al. [44] used stochastic manner to encode rain streaks as a patch based mixture of Gaussian. Li et al.[25] further explored the prior structure of rain streaks and described that they have the repetitive local patterns and multi-scale characteristics. Based on the observation, the authors proposed a multi-scale convolutional sparse coding model that achieves the state-of-the-art deraining performance when the background layer in a video is also finely extracted.

Very recently, deep learning techniques have achieved great success in various low-level vision tasks, such as image denoising [52, 53]

, image super-resolution 

[5, 55, 38], and image deblurring [39, 24]. Recent years have also witnessed the development of deep learning in the deraining task [18]. To remove rains from videos, Liu et al. [30] provided a hybrid rain model and constructed a multi-task learning network architecture that successively accomplished rain degradation classification, rain removal, and background restoration. Further, the authors developed a dynamic routing residue recurrent network [29] for handling dynamically detected rainy videos. Compared with these video deraining methods, the single image deraining task is much more challenging in practice due to the lack of temporal information.

Ii-B Single Image Deraining Methods

Against this single image deraining task, Xu et al. [20] considered the chromatic property of rain streaks and achieved a coarse rain-free image via a guided filter. Kim et al. [22] analyzed the geometric property of rains to detect rain streak regions, and then reconstructed the derained result by executing nonlocal means filtering on the detected region.

Later on, researchers resorted to utilizing domain knowledge to encode rains for helping the deraining task [21, 42]. For example, based on morphological component analysis, Fu et al. [8] considered single image deraining as a signal decomposition problem. Then the authors executed a bilateral filter, dictionary learning, and sparse coding to acquire LFP and HFP for obtaining the derained result. Afterwards, Luo et al. [49] adopted a screen blend model and proposed to utilize high discriminative codes over a learned dictionary to sparsely approximate rain and background layers. To represent multiple orientations and scales of rain streaks, Li et al. [28] designed GMM based patch prior. By employing the sparsity and gradient statistics of rain and background layers, Zhu et al. [57] formulated three regularization terms to progressively extract rain streaks. Afterwards, Gu et al. [11] utilized analysis sparse representation to represent image large-scale structures and synthesis sparse representation to describe image fine-scale textures. Meanwhile, Zhang et al. [14] proposed to learn a set of sparsity based and low rank based convolutional filters for illustrating background and rain layers, respectively. Albeit achieving good performance on certain scenarios, these prior-model-based methods need inevitable time-consuming iterative inferences and always could not finely fit practical diverse rain shapes in real rainy images due to relatively simple while subjective prior assumptions.

More recently, some CNN based DL methods have been raised for the task [27, 51, 41]. Fu et al. [6] first designed the DerainNet to predict clean image. To make training process easier, the authors [7] further proposed DDN to remove rain content in HFP. Later, Zhang et al. [51] proposed a conditional generative adversarial deraining network. Considering the complicatedness of rain streaks, the authors [50]

further incorporated a residual-aware classifier process and designed a rain density-aware multistream dense network. Furthermore, Yang 

et al. [45] reformulated the commonly-used rain model and developed a multi-task architecture to jointly learn binary rain streak map, appearance of rain streaks, and clean background. To achieve better visual quality, the authors further introduced a detail preservation step [46]. Very recently, by repeatedly unfolding a shallow ResNet with a recurrent layer, Ren et al. [34] presented a simple baseline network, PReNet. To make the network better reflect the prior knowledge of rain structures, some beneficial attempts have also been made. For example, Mu et al. [31] proposed to implicitly describe prior structures by data-dependent networks and then formulate them into bi-layer optimization iterations. To alleviate the hard-to-collect-training-sample and overfitting-to-training-sample issues, Wei et al. [43] formulate rain layer prior as GMM and train the backbone, DDN, in a semi-supervised manner. Albeit good performance, these DL methods embed these beneficial prior rain knowledge inside the network architecture [31] or on the loss formulation for training the network [43]. The high flexibility of the nonlinear network mapping might output unexpected mappings deviated from practical rain shapes especially in the testing stages with non-seen rain configurations in training samples. Their generalization performance could thus possibly be negatively influenced. This is the main motivation of this study, to make network extract rain layer, possibly complying with the intrinsic rain structures the previous researches have been explored, from a rainy image.

Fig. 2: Illustration of the proposed SRNet. (a) The whole deraining network architecture. (b) The Encoder-Decoder Network for extracting rain layer at relatively small scale. The skip connections 1 and 2 achieve local residual learning and the skip connection 3 denotes global residual learning [38]. Besides, local residual learning also includes the skip connection in each Resblock [13]. For obtaining relatively medium rain layer and large rain layer , the dilated factor in (b) is replaced with 2 and 3, respectively.

Iii Structural Residual Network for Single Image Rain Removal

Iii-a Basic Rainy Image Model

An input rainy image is denoted as , where and denote its height and width, respectively. The commonly-used rainy image model is:


where and represent the background layer and the rain layer of the image, respectively. The goal of DL based single image derainers is to design rational network architectures to learn non-linear mapping functions from an input rainy image to its background layer or the residual rain layer .

Our aim is to construct a network architecture with its output residual capable of sufficiently representing prior rain structures explored by previous investigations, so as to rectify a rational and interpretable network prediction even for those non-seen and complicated rainy image inputs. Specifically, three known priors have been considered in this network design task. The first is the local-repetitive-pattern prior as proposed in [11] and  [14]. That is, rain streaks always repeatedly appear at different locations over a rainy image with similar local patterns like shape, thickness, and direction. The second is the coefficient-sparsity prior, which describes the fact that the rain streaks are always sparsely scattered over an image [49, 57]. The third is the multi-scale prior [25], which refers to the phenomenon that rain streaks in different distances appear with different sizes in a rainy image since they are pictured from different distances by cameras. Based on this, the single image rainy model (1) can be more elaborately formulated as:


where , , and denote three different scales of rain layer separation, respectively.

We then introduce how to construct a deep network with carefully designed structural residual output finely conveying all the aforementioned prior knowledge of rain streaks.

Iii-B Structural Residual Network Architecture

In this section, we construct a single image deraining network to learn structural residual output. As shown in Fig. 2, the proposed network, called structural residual network (SRNet), mainly consists of three parallel sub-networks with similar structures but different dilated factors (DF), to extract rain layer at different scales. The rain-removed result

is estimated by subtracting the entire residual rain layer

from the input . The design details are described as follows.

Fig. 3: Illustration of the pairwise MaxPooling and MaxUpooling operation with kernel_size as 2[33].
Fig. 4: Rain removal performance comparison on two typical rainy images. (a) Original rainy images. (b) The derained results in the case of “without dilated convolution” (only with small branch). (c) The derained results in the case of “with dilated convolution”.
Fig. 5: Structural rain layers extracted by the SRNet for (a) a synthetic rainy image from Rain100L [46] and (b) a real rainy image from SPA-Data [41]. In both panel, the first row denote (from left to right): rainy image, groundtruth, derained result, and extracted rain layer . The second to forth rows denote: the first column is the output rain layers , , and obtained by our method; The last three columns are the corresponding three representative rain feature maps, i.e., , , and at the scale (*). The figure is better observed by zooming in on a computer screen.

Iii-B1 Convolutional Sparse Coding Structure for Rain Streaks

As displayed in Fig. 2(a), the SRNet first adopts one convolution layer and two Resblocks to extract shallow features . Consistent to the model (2), three encoder-decoder subnetworks are constructed by exploiting the dilated convolutions [48] with different DFs to extract , , and at relatively small, medium, and large scales, respectively. Here we take the extraction of small rain layer as an example and illustrate the encoder-decoder network structure.

As shown in Fig. 2(b), the proposed encoder-decoder subnetwork is symmetrically composed of encoder and decoder parts. In the encoder part, two consecutive Resblocks are to obtain deeper features of the shallow feature input . Each Resblock is followed by one MaxPooling layer with downsampling ratio set as . Symmetrically, two extra Resblocks are stacked in the decoder part and each one is followed by one MaxUnpooling layer for upsampling the feature activations ahead. The MaxUnpooling layer exploits the pooling indices computed on the corresponding MaxPooling layer to execute non-linear unsampling, as shown in Fig. 3. With the symmetrical combination of MaxPooling/MaxUnpooling layers, and the feature fusion operation with successive convolution and convolution, we get the sparse rain feature map , as displayed in Fig. 2(b). Such sparsity finely represents the aforementioned coefficient-sparsity prior, i.e., the sparse locations of rain streaks across the image. Subsequently, by imposing the convolution layer on , small rain layer can then be extracted. Such convolutional coding manner naturally delivers the local-repetitive-pattern prior of rain shapes over the entire image.

From [55] and [15], incorporating skip or dense connection strategy into a deeper model can improve the understanding and representation of image contents. Hence, to fully utilize the shallow feature and propagate the long-distance spatial context information, we introduce the global residual learning [38] into the encoder-decoder network shown as skip connection 3 in Fig. 2(b). Besides, we incorporate diverse local residual learning, including the skip connection inside each Resblock and the skip connections between the encoder part and the decoder part shown as 1 and 2 in Fig. 2(b). In this way, we aim to deepen into stronger feature expressions for better extracting rain structure. Meanwhile, the training process would be made easier [13]. The functions of such MaxUnpooling layer and skip connection settings to the final performance of our network will be validated in our following ablation experiments.

Fig. 6: Performance comparison of all competing methods on a typical test image in Rain100L. The first row: rainy image and derained results obtained by different methods. The second row: groundtruth and rain layers extracted by different methods. PSNR/SSIM results are given below each figure for easy reference. Bold and bold italic indicate top and best performance, respectively.

Iii-B2 Multi-Scale Structure for Rain Streaks

To represent the multi-scale configuration of rain streaks, we simply resort to dilated convolution, which weights pixels with a step size of DF, and enlarges its receptive field without losing resolution [48]. Specifically, we repeatedly construct the proposed encoder-decoder subnetwork with the same kernel size but with three different DFs (1, 2, and 3), as displayed in Fig. 2(a). The three branches thus have their own receptive fields to capture information at different scales. Fig. 4 demonstrates the effect of this multi-scale specification by comparing the rain-removed images restored by our network with and w/o dilated convolutions. One can easily observe that with dilated convolution, the SRNet can acquire larger receptive field that inclines to help remove long rain streaks as well as preserve large-scale image details.

Here, we present a visualization for the structural residual learned by the poposed SRNet to express the rain layer as shown in Fig. 5. The rain layer and rain feature maps (, , and ) on two testing images by our network are demonstrated. It is easy to see obvious differences among , , and with multi-scale characteristics. , , and approximately illustrate the directions, positions, and shapes of rain streaks, implying the convolutional coding mechanism of the proposed method. In such manner, the proposed SRNet is expected to capture the rational rain streaks complying with these pre-known prior structures.

Iii-C Objective Function

As the sensitivity of the human visual system depends on local luminance, contrast, and structure, this can be quantitatively measured by the structure similarity (SSIM) index [56]. The larger the SSIM value is, the better the quality of the image is. To train the SRNet, similar to [34], we simply adopt the negative SSIM as the objective function, written as:


Iii-D Training Details

We use PyTorch


to implement the proposed SRNet, based on a PC equipped with an Intel (R) Core(TM) i7-8700K at 3.70GHZ and one Nvidia GeForce GTX 1080Ti GPU. Since we mainly care about the effectiveness of the network architecture on rain removal, instead of relying on some extra tricks such as designing complicated loss functions and tuning hyper-parameters, we directly adopt the parameter settings of the latest baseline network–PReNet 

[34]. Specifically, the patch size is 100100 and the Adam optimizer[23] with the batch size of 18 is used. The initial learning rate is

and divided by 5 after reaching 30, 50, and 80 epochs. The total epoch is 100. It is worth mentioning that for all datasets in subsequent experiments, these parameter settings are the same. This would show the favorable robustness and generality of our method. Besides, to balance the trade-off between running time and deraining performance on synthetic and real datasets, we simply select depth

and width as the default setting in our experiments, where the depth denotes the times of symmetrical downsampling/upsampling operations, namely, the number of the module Resblock+MaxPooing (Resblock+MaxUnpooing) in Fig. 2(b).

Fig. 7: Performance comparison of all competing methods on a typical test image in Rain12. The first row: rainy image and derained results obtained by different methods. The second row: groundtruth and rain layers extracted by different methods.

Iv Experimental Results

In this section, the effectiveness of the SRNet is verified via comprehensive experiments implemented on synthetic and real datasets by comparing with current state-of-the-art methods, including model based ones: DSC [49]111, GMM [28]222, JCAS [11]333; and DL-based ones: Clear [6]444, DDN [7]555, RESCAN [27]666, PReNet [34]777, SPANet [41]888, JORDER_E [46]999, SIRR [43]101010

Iv-a Experiments on Synthetic Data

Synthetic Datasets. We adopt four benchmark datasets: Rain100L [46], Rain100H [46], Rain1400 [7], and Rain12 [28]. Rain100L has only one type of rain streaks and consists of 200 image pairs for training and 100 ones for evaluation. With five types of rain streak directions, Rain100H is more challenging and contains 1800 image pairs for training and 100 ones for testing. Rain1400 contains 14000 rainy images synthesized from 1000 clean images with 14 kinds of rain streaks, where 12600 rainy images are used as training samples and 1400 ones as testing samples. Like [34], the trained model for Rain100L is utilized to evaluate Rain12 that only includes 12 image pairs. For SIRR, we adopt the real 147 rainy images [43] as unsupervised training samples.

Performance Metrics.

Since the groundtruths of synthetic datasets are available, we provide quantitative comparisons based on two commonly used metrics: peak-signal-to-noise ratio (PSNR) 

[16] and SSIM. As the human visual system is sensitive to the Y channel of a color image in YCbCr space, we compute PSNR/SSIM based on the luminance channel, and the Matlab computation codes are released by  [46].

Evaluation on Rain Removal. From the derained results shown in Fig. 6, it is easy to see that traditional model-based DSC, GMM, and JCAS methods leave distinct rain streaks, and DL-based Clear, DDN, and SIRR also leave evident rain marks. Besides, apart from RESCAN, PReNet, and the proposed SRNet, other competing methods blur image textures obviously. From the comparison of extracted rain layers, it is observed that the result by the SRNet covers less texture. Fig. 7 further compares the deraining results on another test image with diverse rain patterns. As displayed, among all comparison methods, SRNet more evidently removes rain streaks and restores background image.

Fig. 8: Rain removal effect comparison of all competing methods on a typical test image from Rain100H. .
Fig. 9: Performance comparison of all competing methods on a typical test image in Rain1400.
Fig. 10: Generalization performance comparison of all competing methods on a typical real test image from SPA-Data.
Fig. 11: Generalization performance comparison of all competing methods on a typical real test image with short/light rain streaks from Internet-Data.
Fig. 12: Generalization performance comparison of all competing methods on a typical real test image with long/heavy rain streaks from Internet-Data.

Evaluation on Detail Preservation. We select two more difficult samples from Rain100H and Rain1400, respectively, to evaluate the deraining ability of different methods. As depicted in Fig. 8 and Fig. 9, complicated rain patterns adversely degrade the rain removal performance of most methods. However, the proposed SRNet still has advantages over rain removal and detail preservation, and achieves best PSNR and SSIM scores.

Tables I reports the quantitative comparison results of all competing methods on synthetic datasets with diverse, complicated, and different rain types. It is clear that due to the strong fitting ability of CNN, the deraining performance of DL based methods generally outperforms those by conventional prior-model-based methods. Comparatively, the proposed SRNet achieves the best average PSNR and SSIM values on Rain1400 with 14 kinds of rain types. This reflects the better robustness and generality of SRNet.


Datasets Rain100L Rain100H Rain1400 Rain12




Input 26.90 0.8384 13.56 0.3709 25.24 0.8097 30.14 0.8555


DSC[49] 27.34 0.8494 13.77 0.3199 27.88 0.8394 30.07 0.8664


GMM[28] 29.05 0.8717 15.23 0.4498 27.78 0.8585 32.14 0.9145


JCAS[11] 28.54 0.8524 14.62 0.4510 26.20 0.8471 33.10 0.9305


Clear[6] 30.24 0.9344 15.33 0.7421 26.21 0.8951 31.24 0.9353


DDN[7] 32.38 0.9258 22.85 0.7250 28.45 0.8888 34.04 0.9330


RESCAN[27] 38.52 0.9812 29.62 0.8720 32.03 0.9314 36.43 0.9519


PReNet[34] 37.45 0.9790 30.11 0.9053 32.55 0.9459 36.66 0.9610


SPANet[41] 35.33 0.9694 25.11 0.8332 29.85 0.9148 35.85 0.9572
SIRR[43] 32.37 0.9258 22.47 0.7164 28.44 0.8893 34.02 0.9347


JORDER_E[46] 38.56 0.9827 30.50 0.8967 32.00 0.9347 36.69 0.9618


SRNet 37.37 0.9780 30.05 0.9060 32.88 0.9487 36.71 0.9626


TABLE I: PSNR and SSIM comparisons on synthetic datasets. Bold and bold italic indicate the best and performance, respectively.


Methods Input DSC GMM JCAS Clear DDN


PSNR 34.15 34.95 34.30 34.95 32.66 34.70


SSIM 0.9269 0.9416 0.9428 0.9453 0.9420 0.9343




PSNR 34.70 35.08 35.13 34.85 34.34 35.31


SSIM 0.9376 0.9424 0.9443 0.9357 0.9382 0.9448


TABLE II: PSNR and SSIM generalization comparisons on SPA-Data [41].
Fig. 13: (a) Two typical rainy images with different rain types. (b) The corresponding groundtruths. (c) and (d) Derained results and extracted rain layers by SRNet, respectively.
Fig. 14: Generalization performance comparison of different DL models trained on the corresponding synthetic datasets. (a) Real rainy images. (b)-(d) Derained results predicted by SRNet trained on Rain100L, Rain100H, Rain1400, respectively. Bounded boxes indicate the best ones visually.

Iv-B Experiments on Real Data

Real Datasets and Performance Metrics. For real application, evaluating the generalization ability of a single image derainer is the key point. In this section, we evaluate the deraining performance of all competing methods on real rainy images, based on two real datasets from [41] and  [43], respectively. The former is denoted as SPA-Data and contains 1000 image pairs for evaluation, and the other is called Internet-Data and consists of 147 rainy images without groundtruths. For the experiments on SPA-Data, we compute PSNR and SSIM based on the Y channel. While for Internet-Data, we can only provide visual comparison. Specifically, for each one among the DL-based comparison methods, we can obtain three pre-trained models based on the three synthetic datasets: Rain100L, Rain100H, and Rain1400, respectively. For each method, we select the model with the best PSNR/SSIM on SPA-Data to predict the generalization results for real rainy images from SPA-Data. While for Internet-Data, like previous works, the rain-removed result is the one with the best visual quality among the predicted results by the three models.

Evaluation on SPA-Data. Fig. 10 shows the derained results of all comparison methods on a real rainy image from SPA-Data. As seen, when dealing with the images with complex rain shapes, traditional model based DSC, GMM, and JCAS do not work well, which can be explained by that the adopted specific prior assumptions cannot always accurately represent the complicated rain distribution. DL based competing methods also leave obvious rain streaks and even blur image details, such as PReNet. Nevertheless, as compared with other methods, the proposed SRNet achieve better generalization performance.

To objectively evaluate the generalization ability of these methods, we provide the quantitative rain removal performance comparison on SPA-Data listed in Table II. As reported, the most competitive baseline methods on synthetic datasets, such as RESCAN and JORDER_E, cannot perform as well as the traditional model-based JCAS on the real dataset, possibly due to the overfitting-the-training-samples issue. It is worth mentioning that in this case, JCAS obtains a good SSIM as the method utilizes the complementary of ASR and SSR to finely describe texture structures. Although SPANet has relatively inferior deraining effect on synthetic datasets, its generalization performance is competitive. Observing the comparison results presented in Table I and Table II, it is easy to see that among all comparison methods, the proposed SRNet is a competing single image derainer as it achieves satisfactory rain removal performance on both synthetic and real datasets. This confirms the effectiveness of structural residual design of our network architecture and the fine generalization ability of SRNet.

Evaluation on Internet-Data. Furthermore, we select another two hard samples from Internet-Data with complicated rain distribution. Fig. 11 and Fig. 12 demonstrate the corresponding generalization comparison results. From these figures, similar to the experiment analysis on SPA-Data, traditional model-based methods always leave severe rains, and other DL based methods still have obvious rain marks and blur image details. Comparatively, the proposed SRNet has evident superior performance in rain removal and detail preservation.

Iv-C Rain Layer Extraction Evaluation on SRNet

Here we select several rainy images with different rain shapes from aforementioned datasets to further evaluate the effectiveness of the proposed SRNet on extracting rain layer. As shown in Fig. 13, when dealing with complicated rainy images with diverse rain streaks (light/heavy density, short/long shape, different directions), the proposed method can finely extract rain layer and preserve background details well.

Iv-D Ablation Studies

The Effect of Training Datasets. Based on some typical real rainy images, Fig. 14 shows the derained results predicted by SRNet trained on the three synthesized datasets, including Rain100L, Rain100H, and Rain1400. For each input rainy image, we use a box to indicate the best recovery result visually. It can be observed that the model trained on Rain100L performs well in the case of light/thin rain streaks; the trained model on Rain100H is applicable to heavy/accumulated rain types, and for Rain1400, it is suitable for long and thin ones.

Network Modules Analysis. Here we quantitatively analyze the importance of different modules of the proposed SRNet, as listed in Table III. We train all the variants on Rain100L and compare their generalization ability on SPA-Data. , , , and denote four variants of only using the small scale branch (DF=1) in Fig. 2(a). and are basic Resnet-like networks, only consisting of Resblocks. and include symmetrical downsampling and unsampling layers following the corresponding Resblocks. is the default SRNet. is a variant that SRNet adopts weight sharing among three parallel branches, which shows that the SRNet can notably reduce network parameters by weight sharing without unsubstantial degradation in generalization performance. It is worth mentioning that even only adopts small scale, it outperforms most other competing methods on real deraining performance as shown in Table II, which corroborates the rationality of the proposed encoder-decoder network. The versus between every two variants and the corresponding functioning module is concluded as Table IV.






Resblock w/o pooling


Encoder-Decoder Net


Bilinear unsampling




Global residual learning


Weight sharing


w/o Weight sharing


SPA-Data PSNR 35.06 35.15 35.20 35.23 35.22 35.31
SSIM 0.9414 0.9437 0.9438 0.9441 0.9444 0.9448


TABLE III: Network module analysis of the proposed SRNet.


Versus Winner Effective Module


vs Global residual learning


vs Encoder-Decoder Net


vs MaxUpooling


vs Multi-Scale


TABLE IV: Versus among the variants of SRNet given in Table III.

Iv-E Extension to Real Video Deraining

Finally, we evaluate the deraining ability of the SRNet on real rainy videos without groundtruth, by comparing with current state-of-the-art video deraining methods, including the traditional model-based Garg et al. [10], Kim et al. [19], Jiang et al. [17], Ren et al. [35], Wei et al. [44], Li et al. [25], and the DL-based Liu et al. [30].

Fig. 15 and Fig. 16 present the visual comparison results based on two real rainy frames: one is captured by surveillance systems on a street with complex moving objects, and the other is obtained at night. From Fig. 15, it is easily seen that the model-based video deraining methods, including Garg et al.’s, Kim et al.’s, Jiang et al.’s, Wei et al.’s, and Li et al.’s, cause different degrees of artifacts at the location of the moving car, and the DL based Liu et al.’s method leaves obvious rain streaks. Even not utilizing temporal information among multi-frames, the proposed single image deraining method, SRNet, still performs well on removing rain streaks and preserving background textures. From Fig. 16, SRNet has the ability to detect obvious rain streaks. Although some model-based video deraining methods can remove more rain streaks than SRNet, they need many frames and thus have unfavorable real-time performance. It is noteworthy that as compared with the DL-based Liu et al.’s video deraining method, SRNet can also preserve relatively more image details, like trunks.

Fig. 15: Deraining performance of competing video deraining methods and SRNet on a real video frame with complex moving objects.
Fig. 16: Deraining performance of competing video deraining methods and SRNet on a real video frame captured at night.

V Conclusions and Future Work

In this paper, we have taken into account the intrinsic prior structures of rain streaks, and designed a specific structural residual network for rain layer removal from a single image. By visualizing the extracted rain layer and sparse rain feature maps for synthetic and real test images, we have validated the working mechanism underlying the proposed structural residual network. Comprehensive experiments have demonstrated the superiority of the proposed method over current state-of-the-art single image derainers in terms of both robustness and generalization capability.

Although we have fairly evaluated the generalization performance of all competing methods based on SPA-Data, the dataset relies on human judgments and it cannot cover all the complicated patterns of rains in real-world. Besides, collecting a large number of really real rainy/clean image pairs is extremely time-consuming and cumbersome. To alleviate these problems, in the future, we will make further efforts for employing such network designing idea into unsupervised or at least semi-supervised scenarios in our future investigations.


  • [1] C. H. Bahnsen and T. B. Moeslund (2018) Rain removal in traffic surveillance: does it matter?. IEEE Transactions on Intelligent Transportation Systems 20 (8), pp. 2802–2819. Cited by: §I.
  • [2] P. C. Barnum, S. Narasimhan, and T. Kanade (2010) Analysis of rain and snow in frequency space.

    International journal of computer vision

    86 (2-3), pp. 256.
    Cited by: §II-A.
  • [3] Y. L. Chen and C. T. Hsu (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1968–1975. Cited by: §II-A.
  • [4] X. Ding, L. Chen, X. Zheng, H. Yue, and D. Zeng (2016) Single image rain and snow removal via guided l0 smoothing filter. Multimedia Tools and Applications 75 (5), pp. 2697–2712. Cited by: §I.
  • [5] C. Dong, C. C. Loy, K. He, and X. Tang (2015) Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38 (2), pp. 295–307. Cited by: §II-A.
  • [6] X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Transactions on Image Processing 26 (6), pp. 2944–2956. Cited by: §I, §II-B, TABLE I, §IV.
  • [7] X. Fu, J. Huang, D. Zeng, H. Yue, X. Ding, and J. Paisley (2017) Removing rain from single images via a deep detail network. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    pp. 3855–3863. Cited by: §I, §II-B, §IV-A, TABLE I, §IV.
  • [8] Y. Fu, L. Kang, C. Lin, and C. Hsu (2011) Single-frame-based rain removal via image decomposition. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1453–1456. Cited by: §II-B.
  • [9] K. Garg and S. K. Nayar (2004) Detection and removal of rain from videos. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. I–I. Cited by: §II-A.
  • [10] K. Garg and S. K. Nayar (2007) Vision and rain. International Journal of Computer Vision 75 (1), pp. 3–27. Cited by: §II-A, §IV-E.
  • [11] S. Gu, D. Meng, W. Zuo, and Z. Lei (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1708–1716. Cited by: §I, §I, §II-B, §III-A, TABLE I, §IV.
  • [12] K. He, J. Sun, and X. Tang (2010) Guided image filtering. IEEE transactions on pattern analysis and machine intelligence 35 (6), pp. 1397–1409. Cited by: §I.
  • [13] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: Fig. 2, §III-B1.
  • [14] Z. He and V. M. Patel (2017) Convolutional sparse and low-rank coding-based rain streak removal. In IEEE Winter Conference on Applications of Computer Vision, pp. 1259–1267. Cited by: §I, §II-B, §III-A.
  • [15] G. Huang, Z. Liu, L. V. Der Maaten, and K. Q. Weinberger (2017) Densely connected convolutional networks. pp. 2261–2269. Cited by: §III-B1.
  • [16] Q. Huynh-Thu and M. Ghanbari (2008) Scope of validity of psnr in image/video quality assessment. Electronics Letters 44 (13), pp. 800–801. Cited by: §IV-A.
  • [17] T. Jiang, T. Huang, X. Zhao, L. Deng, and Y. Wang (2017)

    A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors

    In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4057–4066. Cited by: §IV-E.
  • [18] C. Jie, C. H. Tan, J. Hou, L. P. Chau, and L. He (2018) Robust video content alignment and compensation for rain removal in a cnn framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6286–6295. Cited by: §II-A.
  • [19] K. Jin-Hwan, S. Jae-Young, and K. Chang-Su (2015) Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing 24 (9), pp. 2658–2670. Cited by: §II-A, §IV-E.
  • [20] X. Jing, Z. Wei, L. Peng, and X. Tang (2012) Removing rain and snow in a single image using guided filter. In IEEE International Conference on Computer Science and Automation Engineering, Vol. 2, pp. 304–307. Cited by: §I, §II-B.
  • [21] L. W. Kang, C. W. Lin, and Y. H. Fu (2012) Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing 21 (4), pp. 1742–1755. Cited by: §II-B.
  • [22] J. H. Kim, C. Lee, J. Y. Sim, and C. S. Kim (2014) Single-image deraining using an adaptive nonlocal means filter. In IEEE International Conference on Image Processing, pp. 914–917. Cited by: §II-B.
  • [23] D. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. Computer Science. Cited by: §III-D.
  • [24] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas (2018)

    Deblurgan: blind motion deblurring using conditional adversarial networks

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8183–8192. Cited by: §II-A.
  • [25] M. Li, Q. Xie, Q. Zhao, W. Wei, S. Gu, J. Tao, and D. Meng (2018) Video rain streak removal by multiscale convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6644–6653. Cited by: §I, §II-A, §III-A, §IV-E.
  • [26] S. Li, I. B. Araujo, W. Ren, Z. Wang, E. K. Tokuda, R. H. Junior, R. Cesar-Junior, J. Zhang, X. Guo, and X. Cao (2019) Single image deraining: a comprehensive benchmark analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3838–3847. Cited by: §I.
  • [27] X. Li, J. Wu, Z. Lin, H. Liu, and H. Zha (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In Proceedings of the European Conference on Computer Vision, pp. 254–269. Cited by: §I, §II-B, TABLE I, §IV.
  • [28] Y. Li (2016) Rain streak removal using layer priors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2736–2744. Cited by: §I, §II-B, §IV-A, TABLE I, §IV.
  • [29] J. Liu, W. Yang, S. Yang, and Z. Guo (2018) D3R-net: dynamic routing residue recurrent network for video rain removal. IEEE Transactions on Image Processing 28 (2), pp. 699–712. Cited by: §II-A.
  • [30] J. Liu, W. Yang, S. Yang, and Z. Guo (2018) Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3233–3242. Cited by: §II-A, §IV-E.
  • [31] P. Mu, J. Chen, R. Liu, X. Fan, and Z. Luo (2019) Learning bilevel layer priors for single image rain streaks removal. IEEE Signal Processing Letters 26 (2), pp. 307–311. Cited by: §II-B.
  • [32] W. Park and K. Lee (2008)

    Rain removal using kalman filter in video

    In International Conference on Smart Manufacturing Application, pp. 494–497. Cited by: §II-A.
  • [33] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer (2017) Automatic differentiation in pytorch. Cited by: Fig. 3, §III-D.
  • [34] D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng (2019) Progressive image deraining networks: a better and simpler baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3937–3946. Cited by: §I, §II-B, §III-C, §III-D, §IV-A, TABLE I, §IV.
  • [35] W. Ren, J. Tian, H. Zhi, A. Chan, and Y. Tang (2017) Video desnowing and deraining based on matrix decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4210–4219. Cited by: §II-A, §IV-E.
  • [36] M. S. Shehata, J. Cai, W. M. Badawy, T. W. Burr, M. S. Pervez, R. J. Johannesson, and A. Radmanesh (2008) Video-based automatic incident detection for smart roads: the outdoor environmental challenges regarding false alarms. IEEE Transactions on Intelligent Transportation Systems 9 (2), pp. 349–360. Cited by: §I.
  • [37] C. E. Smith, C.A. Richards, S.A. Brandt, and N.P. Papanikolopoulos Visual tracking for intelligent vehicle-highway systems. IEEE Transactions on Vehicular Technology 45 (4), pp. 744–759. Cited by: §I.
  • [38] Y. Tai, J. Yang, and X. Liu (2017) Image super-resolution via deep recursive residual network. pp. 2790–2798. Cited by: Fig. 2, §II-A, §III-B1.
  • [39] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia (2018) Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174–8182. Cited by: §II-A.
  • [40] A. Tripathi and S. Mukhopadhyay (2012) Video post processing: low-latency spatiotemporal approach for detection and removal of rain. IET image processing 6 (2), pp. 181–196. Cited by: §II-A.
  • [41] T. Wang, X. Yang, K. Xu, S. Chen, Q. Zhang, and R. W. Lau (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12270–12279. Cited by: §I, §I, §II-B, Fig. 5, §IV-B, TABLE I, TABLE II, §IV.
  • [42] Y. Wang, S. Liu, C. Chen, and B. Zeng (2017) A hierarchical approach for rain or snow removing in a single color image. IEEE Transactions on Image Processing 26 (8), pp. 3936–3950. Cited by: §II-B.
  • [43] W. Wei, D. Meng, Q. Zhao, Z. Xu, and Y. Wu (2019)

    Semi-supervised transfer learning for image rain removal

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3877–3886. Cited by: §II-B, §IV-A, §IV-B, TABLE I, §IV.
  • [44] W. Wei, L. Yi, Q. Xie, Q. Zhao, D. Meng, and Z. Xu (2017) Should we encode rain streaks in video as deterministic or stochastic?. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2516–2525. Cited by: §II-A, §IV-E.
  • [45] W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan (2017) Deep joint rain detection and removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366. Cited by: §I, §II-B.
  • [46] W. Yang, R. T. Tan, J. Feng, J. Liu, S. Yan, and Z. Guo (2019) Joint rain detection and removal from a single image with contextualized deep networks. IEEE Transactions on Pattern Analysis and Machine Intelligence PP (99), pp. 1–1. Cited by: §I, §II-B, Fig. 5, §IV-A, §IV-A, TABLE I, §IV.
  • [47] R. Yasarla and V. M. Patel (2020) Confidence measure guided single image de-raining. IEEE Transactions on Image Processing 29, pp. 4544–4555. Cited by: §I.
  • [48] F. Yu and V. Koltun (2015) Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122. Cited by: §I, §III-B1, §III-B2.
  • [49] L. Yu, X. Yong, and J. Hui (2015) Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405. Cited by: §I, §I, §II-B, §III-A, TABLE I, §IV.
  • [50] H. Zhang and V. M. Patel (2018) Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 695–704. Cited by: §II-B.
  • [51] H. Zhang, V. Sindagi, and V. M. Patel (2019) Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology. Cited by: §II-B.
  • [52] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang (2017) Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing 26 (7), pp. 3142–3155. Cited by: §II-A.
  • [53] K. Zhang, W. Zuo, and L. Zhang (2018) FFDNet: toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing 27 (9), pp. 4608–4622. Cited by: §II-A.
  • [54] X. Zhang, H. Li, Y. Qi, W. K. Leow, and T. K. Ng (2006) Rain removal in video by combining temporal and chromatic properties. In IEEE International Conference on Multimedia and Expo, pp. 461–464. Cited by: §II-A.
  • [55] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu (2018) Residual dense network for image super-resolution. pp. 2472–2481. Cited by: §II-A, §III-B1.
  • [56] W. Zhou, B. Alan Conrad, S. Hamid Rahim, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13 (4), pp. 600–612. Cited by: §III-C.
  • [57] L. Zhu, C. W. Fu, D. Lischinski, and P. A. Heng (2017) Joint bi-layer optimization for single-image rain streak removal. In Proceedings of the IEEE international conference on computer vision, pp. 2526–2534. Cited by: §I, §II-B, §III-A.