I Introduction
As living standard improves rapidly, more and more people nowadays like to go travelling and they often take photos of themselves or the landscape when they are out. And with the booming of social media such as Facebook and YouTube, it nearly becomes a routine for some people to photograph or video their everyday life and share them with others through Internet. However, many photos are captured under lowlight circumstance due to backlight, under exposure or dark environment. Although the improvement of skills, equipments, and conditions could to some extent avoid that, it is still inevitable to take photos with undesirable quality, let alone the massive images with the problem acquired in the past. These photos so called lowlight images suffer from low definition, low contrast and much noise. Postprocessing techniques are required to enhance the visual quality of these images.
It is the most intuitive and simplest way to directly amplifying the illumination of a lowlight image. But this operation results in some other problems like saturating bright areas and losing details. Histogram equalization (HE) based methods [1] flatten the histogram and stretch the dynamic range of the intensity, alleviating the above problems. But the results of these methods may be under or overenhanced with much noise. Some researchers [2, 3] noticed the similarity between haze images and the inverted lowlight images. Thus they applied dehazing methods to deal with lowlight image. With respect to the intensive noise, Li et. al. [2] tried to eliminate the influence of noise via BM3D [4] after enhancement. A jointbilateral filter is applied in [3] to suppress the noise after the enhancement.
Retinexbased methods consider the scene in human s eyes as the product of reflectance and illumination layers. Logarithmic transformation can simply the multiplication. However, A recent work [5] comes up with the opinion that logarithmic transformation is not appropriate despite being widely adopted. To cover the shortage of logarithmic form they give a weighted variational model which estimate both the reflectance and the illumination. The model shows surprising results but the noise is quite observable in the results, especially when there is much noise in original images. Another work [6] pays attention to estimate the strengthened illumination map by attaching a coefficient matrix. Although this method obtains impressive results, it also generates the problem of overenhancement and losing details in bright areas. Besides, since the unprocessed reflectance contains much noise, the enhanced image often has noise and an extra denoising procedure via BM3D [4] is also needed. Yue et. al. [7] concentrated on intrinsic image decomposition and introduces constraints on both reflectance and illumination layers. But similarly, they do not take noise as a component or an influence factor in the decomposition procedure.
In our work, we consider noise as a nonnegligible factor in Retinex based decomposition. Thus lowlight enhancement should be aware that eliminating noise must be simultaneously proceeded while enhancing the illumination, not in a separate way. Based on that, we propose an integrative method to simultaneously enhance the images and suppress the noise. Our method estimates both the illumination and the reflectance but in a successive sequence. That is, we first estimate the illumination map, independent from the reflectance map. Then we refine the reflectance on the basis of both the refined illumination and the original image. Given that noise exists in the source image, after we extract the smooth illumination map, most noise is left in the reflectance. Therefore, we use weighted matrices to restrict noise when embellishing the reflectance. We argue that estimating the illumination and reflectance simultaneously using iterative method may introduce more noise to the expected illumination map because noise is often observed in the reflectance image in the classic Retinex decomposition. So a sequential estimation method can obtained a more purified illumination and as a consequence, a better reflectance with noise limited to the minimum. After obtaining the preferred illumination and reflectance, the final enhancement result is generated by combining the reflectance and the Gamma corrected illumination.
The rest of this paper is organized as follows. The proposed approach is elaborated in Section II. Experimental results are presented in Section III. Finally we draw a conclusion in Section IV.
Ii The Joint LowLight Enhancement and Denoising Method
Iia Overview
The classic Retinex model decomposes images into reflectance and illumination as , where is the observed image, and represent the reflectance and the illumination of the image. The operator denotes the elementwise multiplication.
Knowing that low light may introduce much noise to the image and enhancing the picture inevitably intensifies the noise at the same time, we hold the view that the classic Retinex model should be modified with a noise term as follows:
(1) 
Many methods focus on the illumination component and simply take as the obtained reflectance, which actually keeps most unpleasant noise intact in the reflectance image for .
Thus, those methods always lead to noisy results and often require an extra denoising procedure. However, this may cause the missing of some critical details in the image. Besides, some methods use term in their equations, and in order to calculate both and simultaneously, they iteratively update each variable while regarding the other variables as constants. In other words, is calculated on the basis of the previous result of in every iteration. But we argue that, during these procedures the noise, which is often observed in the reflectance image, continuously impairs the expected illumination map .
For those reasons, we propose a new optimization method here that consider both reflectance and illumination but calculate them in a way more respective, as well as seeing noise as one of the affecting factors. Fig. 1 shows the framework of our method.
IiB The Sequential Estimation
As discussed above, we choose to propose sequential equations to acquire the most undisturbed illumination and the most preferred reflectance :
(2) 
(3) 
where , , and are the coefficients that control the importance of different terms. and represent the Frobenius norm and norm, respectively. In addition, is the first order differential operator, denotes the elementwise multiplication, is a weight matrix related to the observed image , and is the adjusted gradient of . The role of each term in (2) and (3) are interpreted below:

takes care of the fidelity between the initial illumination map and the refined one ;

corresponds to the total variation sparsity and considers the piecewise smoothness of the illumination map ;

constrains the fidelity between the observed image and the recomposed one . In other words, and ;

enforces the spatial smoothness on the reflectance accommodatively;

minimizes the distance between the gradient of the reflectance and that of the observed image , so that the contrast of the final result can be strengthened.
In this paper, we assume that, for color images, three channels share the same illumination map. Therefore, we generally set the initial illumination map as channel of the input image.
With the hope of getting a preferable reflectance whose gradients are smooth in homogeneous areas while undamaged at edges, we reasonably set as follows:
(4) 
The modified term enforces the spatial smoothness on the reflectance as well, but the extent at different position of the image is under the control of . This is effective especially when there is intensive, largescale noise in the original image.
As for the matrix , it is an adjusted version of , designed to amplify the reflectance while restrain noise. The formulation of is given as follows:
(5) 
where controls the degree of the amplification; controls the amplification rate of different gradients; is the threshold that filters small gradients.
By suppressing small gradients first, this equation minimizes the possible noise. And then strengthen the overall gradients with alterable proportions.
For each observed image, matrix , and only need to be calculated once.
IiC The Solution
Inspired by lowlight image enhancement via illumination map estimation (LIME) [6], we use the alternative to approximate . As a result, the approximate problem to (2) can be written as follows:
(6) 
This change does not influence the result much because according to the first term , the gradients of should also be similar to those of . For convenience, we put (6) in a simpler from, where represents :
(7) 
As can be observed, (7) only involves quadratic terms. Thus, by differentiating (7) with respect to and setting the derivative to 0, the problem can be directly figured out by solving the following:
(8) 
where
is the identity matrix with proper size. And
contains and , which are the Toeplitz matrices from the discrete gradient operators with forward difference. Further,is the vectorized version of
and the operator Diag() is to construct a diagonal matrix using vector . Then we can easily solve it to obtain the evaluated .Similarly, by differentiating (3) with respect to and setting the derivative to 0, we have the following equation:
(9) 
After the estimation of the illumination and the reflectance , the gamma correction operation is applied in order to adjust the illumination. And the final enhancement result is generated by gamma correction as , where is the normalized , and is empirically set as 2.2.
Iii Experimental Results
All experiments are performed on MATLAB R2017a with 4G RAM and Intel Core i54210H CPU @2.90GHz. In our experiment the parameters , and in (2) and (3)are empirically set as 0.007, 0.001 and 0.016. and are set to be 10 and is set to be 6. In general cases, this setting performs pleasant outcome. Our test images come from dataset provided by authors of [5] and [6]. More experimental results and code can be found in the authors’ webpage ^{1}^{1}1https://github.com/tonghelen/JEDMethod .
First, we compare the illumination, reflectance and outcome images of our method and LIME [6] with details. From Fig. 2 we can see that our illumination maps contain more details than LIME and our reflectance maps generally have more color information. As can be seen in Fig. 2 (e), our results successfully reserve the local details but erase the noise.
To evaluate the enhancing effectiveness of our proposed method, we compare it with conventional histogram equalization (HE) and stateoftheart Retinex based enhancement methods, i.e. simultaneous reflectance and illumination estimation (SRIE) [5], naturalness preserved enhancement algorithm (NPEA) [8], and LIME [6]. From Fig. 3 we can find that HE, SRIE and NEPA do not obviously enhance the image and have lots of noise. LIME often overenhances the image and therefore loses some details in the bright area. Only our method keeps the details of grass in the first image.
To demonstrate the denoising effectiveness of our proposed method, we compare our results with the results of HE, LIME [6], NPEA [8] and probabilistic method for image enhancement (PIE) [9]. All methods except our proposed method are followed by an extra denoising procedure via BM3D [4]. We can find from Fig. 4 that those enhancement works either make their denoising procedure afterwards difficult and less effective, or often come up with detailloosing and blurring problems like PIE. Compared to these methods, our method shows strong advantages in both lowlight enhancement and denoising.
Iv Conclusion
In this paper we discuss the existing problem of noise in mainstream methods of lowlight enhancement domain. And we argue that existing methods either ignore this issue or do not handle it well. According to that, we constructively present a joint lowlight enhancement and denoising method based on sequential decomposition method. By intentionally limiting noise to the minimum, we can obtain highquality images finally. Extensive experimental results demonstrate the effectiveness of our method.
References
 [1] S. M. Pizer, R. E. Johnston, J. P. Ericksen, B. C. Yankaskas, and K. E. Muller, “Contrastlimited adaptive histogram equalization: speed and effectiveness,” in [1990] Proceedings of the First Conference on Visualization in Biomedical Computing, May 1990, pp. 337–345.
 [2] L. Li, R. Wang, W. Wang, and W. Gao, “A lowlight image enhancement method for both denoising and contrast enlarging,” in 2015 IEEE International Conference on Image Processing (ICIP), Sept 2015, pp. 3730–3734.
 [3] L. Zhang, P. Shen, X. Peng, G. Zhu, J. Song, W. Wei, and H. Song, “Simultaneous enhancement and noise reduction of a single lowlight image,” IET Image Processing, vol. 10, no. 11, pp. 840–847, 2016.
 [4] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transformdomain collaborative filtering,” IEEE Transactions on Image Processing, vol. 16, no. 8, pp. 2080–2095, Aug 2007.

[5]
X. Fu, D. Zeng, Y. Huang, X. P. Zhang, and X. Ding, “A weighted variational
model for simultaneous reflectance and illumination estimation,” in
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
, June 2016, pp. 2782–2790.  [6] X. Guo, Y. Li, and H. Ling, “LIME: Lowlight image enhancement via illumination map estimation,” IEEE Transactions on Image Processing, vol. 26, no. 2, pp. 982–993, Feb 2017.
 [7] H. Yue, J. Yang, X. Sun, F. Wu, and C. Hou, “Contrast enhancement based on intrinsic image decomposition,” IEEE Transactions on Image Processing, vol. 26, no. 8, pp. 3981–3994, Aug 2017.
 [8] S. Wang, J. Zheng, H. M. Hu, and B. Li, “Naturalness preserved enhancement algorithm for nonuniform illumination images,” IEEE Transactions on Image Processing, vol. 22, no. 9, pp. 3538–3548, Sept 2013.
 [9] X. Fu, Y. Liao, D. Zeng, Y. Huang, X. P. Zhang, and X. Ding, “A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 4965–4977, Dec 2015.
Comments
There are no comments yet.