Depth and Reflection Total Variation for Single Image Dehazing

01/22/2016 ∙ by Wei Wang, et al. ∙ Chongqing University 0

Haze removal has been a very challenging problem due to its ill-posedness, which is more ill-posed if the input data is only a single hazy image. In this paper, we present a new approach for removing haze from a single input image. The proposed method combines the model widely used to describe the formation of a haze image with the assumption in Retinex that an image is the product of the illumination and the reflection. We assume that the depth and reflection functions are spatially piecewise smooth in the model, where the total variation is used for the regularization. The proposed model is defined as a constrained optimization problem, which is solved by an alternating minimization scheme and the fast gradient projection algorithm. Some theoretic analyses are given for the proposed model and algorithm. Finally, numerical examples are presented to demonstrate that our method can restore vivid and contrastive hazy images effectively.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 9

page 10

page 11

page 12

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

In foggy and hazy days, the light reflected from an object is usually absorbed and scattered by the turbid medium such as particles and water droplets in the atmosphere before it reaches the camera lens. Moreover, the light captured by the camera lens is often blended with the atmospheric light. This inevitably causes outdoor images taken in bad weather (e.g., foggy or hazy) to lose intensity contrast and color fidelity. Moreover, because most automatic systems strongly depend on the definition of the input images, they usually fail to work on the hazy images. Therefore, haze removal (or dehazing) is very important for the computer vision applications, such as remote sensing

[13], video surveillance systems [27], and so on.

In computer vision and computer graphics, the model widely used to describe formation of a hazy image is that a hazy scene appearance is a weighted sum of the attenuated scene radiance and the transmitted atmospheric light whose extents depend on the scene distance to the camera lens (depth information) [29]. However, the depth information and the value of the atmospheric light are usually unknown in practice. Therefore, haze removal is a very challenging problem. The problem will be more ill-posed if the input data is only a single hazy image [8].

In the literature, there exist mainly two classes of approaches for removing haze. A class of approaches is dependent on multiple images or additional information. Polarization methods [20][21] remove haze by using the difference of the polarization characteristics from two or more images of the same scene taken with different polar angles. In [17][12][16], depth-based methods are proposed, which obtain the depth information from the user inputs or from known 3D models. These methods are effective for image dehazing, but they are somewhat impractical because both multiple images and a 3D geometrical model are usually difficult to acquire in practice.

Another class of approaches is to only use a single image to remove haze. This class can also be divided into two categories: filter methods and energy methods. The filter methods benefit from new image models or stronger priors. Fattal [4]

recovered the haze-free scene contrast, under the assumption that the optical transmission and surface shading are not locally correlated to the estimation of the albedo of the scene and the optical transmission. Though Fattal’s assumption is physically reasonable, it is invalid when the haze is heavy or the assumption fails. Tan

[22] removed haze by maximizing the local contrast of the restored image based on the observation that haze-free images have higher contrast than hazy images. Their results are greatly enhanced, but tend to be over-saturated, and the results may not be physically valid. Tarel and Hautière [23] introduced a fast algorithm to dehaze both color images and gray level images by using an edge preserving median filter. He et al. [8] proposed a dark channel prior (DCP) based on the observation that in most of the local regions it is very often that some pixels have very low intensity in at least one color (RGB) channel. Using this prior, they can recover high-quality haze-free images. However, this method has a high computational complexity in the soft matting process. After the DCP was proposed, the approaches based on this prior develop rapidly. In [9], He et al. proposed a guided image filtering in place of soft matting to speed up the process of dehazing. In [26], based on a physical model and the DCP, Wang et al. proposed a single image dehazing method which chooses the atmospheric light founded on a Variogram. Chen and Huang [2] presented a new image haze removal approach based on Fisher’s linear discriminant-based dual DCP scheme in order to solve the problems associated with the presence of localized light sources and color shifts, and thereby achieved effective restoration. In [3], Choi et al. proposed a referenceless perceptual fog density prediction model based on natural scene statistics (NSS) and fog aware statistical features. This model only makes use of measurable deviations from statistical regularities observed in natural foggy and fog-free images, which not only assesses the performance of defogging algorithms designed to enhance the visibility of foggy images, but also is well suited for image defogging. In [10], Huang et al. presented a novel Laplacian-based visibility restoration approach to solve inadequate haze thickness estimation and alleviate color cast problems. In [30], Dai et al. proposed an adaptive sky detection and preservation method to dehaze the images. The method detects the sky by hard or smooth threshold of the sky area which selects the threshold value as a function of the histogram of the image to process.

The energy methods usually propose an energy functional that should be minimized or maximized and derive a numerical scheme. Nishino et al. [19] proposed a Bayesian probabilistic dehazing method that jointly estimates the scene albedo and depth from a single foggy image by fully leveraging their latent statistical structures. Their method is effective for dehazing images but needs to tune a lot of parameters that have great influence on the final results. In [5], Fang et al. proposed a new energy model for dehazing and denoising simultaneously based on a windows adaptive DCP. In [6], Fattal derived a local formation model that explains the color-lines which means pixels of small image patches typically exhibit a 1D distribution in RGB color space and used it for recovering the scene transmission based on the lines’ offset from the origin. In [24], Wang et al. presented a multiscale depth fusion (MDF) method for defog from a single image. The fusion is formulated as an energy minimization problem that incorporates spatial Markov dependence. They got the depth map by minimizing the nonconvex potential in the random field which was solved by an alternate optimization algorithm. In [28]

, Zhu et al. proposed a Color Attenuation Prior to dehaze images. They recovered the depth information via creating a linear model for modeling the scene depth whose parameters are learned by a supervised learning method. In

[7], Galdran et al. extended a perception-inspired variational framework for single image dehazing by substituting the mean of the clean image for the value of the gray-world hypothesis and adding a set of new terms to the energy functional for maximizing the interchannel contrast. In [14]

, Lai et al. derived the optimal transmission map directly from the haze model under two scene priors including locally consistent scene radiance and context-aware scene transmission. They introduced theoretic and heuristic bounds of scene transmission and incorporated the two scene priors to formulate a constrained minimization problem which solved by quadratic programming.

In this paper, we propose a new energy approach for removing haze from a single input image. Our model is a constrained total variation model which is deduced by combining the model widely used to describe the formation of a haze image with the assumption in Retinex that an image is the product of the illumination and the reflection. Then we use an alternating minimization scheme and the fast gradient projection (FGP) algorithm [1] to solve the proposed model and give some theoretical analysis for our model and the algorithm. At last, some experimental results are presented to show the effectiveness of our method.

The rest of our paper is organized as follows. In Section 2, we deduce the proposed model and show the existence and uniqueness of solution for the model. In Section 3, we present the algorithm to solve our model and investigate its convergence. In Section 4, we give the experimental results and conclude in Section 5.

Ii The proposed model

In computer vision and computer graphics, the formulation widely used to describe the hazy image is as follows [29]:

(1)

where is the observed intensity defined on the image domain , is the radiance of the surface, is a constant representing the globally atmospheric light, and is the scene transmission describing the portion of the light that is not scattered and reaches the camera lens. When the atmosphere is homogenous, the transmission can be formulated as [29]:

(2)

where is the positive scattering coefficient of the atmosphere and is the depth of scene which is also positive.

According to the assumption widely used in Retinex [11][18][25][4][6], the surface radiance can be written as a product of the reflectance and the illumination:

(3)

where and is the reflectance and the illumination, respectively. The reflectance is related to the physical characteristics of the surface. For the following theoretic analysis, we assume that a.e. (almost everywhere). Based on the physical characteristics of the material object, the constraint a.e. is practical.

Substituting (3) into (1), we have

(4)

Because most hazy images are landscapes of outdoor scenarios, we make the following assumption:

(5)

Since a.e., by (4) and (5) we obtain

(6)

Equation (6) implies

Because the atmospheric light is a constant, we have

where is a constant. By the way, by experiments we found that the proposed model is insensitive to the value of (or ), the assumption (5) is valid by choosing larger value of .

Equation (4) can be rewritten as

(7)

Let . Because of (5), we have . In particular, if , then . In order to handle the product expression in (7), we convert the product to the logarithmic form. Let Then, Equation (7) is equivalent to

Besides, since and , we have , and so , . Therefore, the constraints and , a.e. need to be added into the proposed model.

It should be pointed out that Equation (7) has the similar form with Equation (8) in [19], but the deriving processes are different. We derive Equation (7) by merging the Retinex theory into the haze imaging model (1), while Nishino et al. [19] derive their model by rewriting the Koschmieder’s law. The constraint is very important for our variational method (see Figure 10). However, this constraint cannot be obtained by Nishino et al.’ method of deriving Equation (8) in [19].

We further make the following assumptions before presenting our model:

  • The depth function and the reflection function are piecewise smooth in . Thus and the regularization terms are defined by and .

  • The sum of the depth function and the reflection function is close to . Thus, the fidelity term is given by .

Collecting the above two assumptions into one expression, we propose the following energy functional for dehazing:

(8)

where and are regularization parameters and is an arbitrary small constant.

The term is only used for the proof of the uniqueness of the minimizer. Without this term, a constant can be added to variable and subtracted to variable without a change of the value of the energy functional. Numerically, there is no difference for our model with being a very small value (such as ) and being zero. Therefore, we always set in numerical implementation.

The proposed model is thus given by the constrained optimization problem:

(9)

We first show the existence of solution for the above optimization problem.

Theorem 1

The functional is strictly convex in .

The proof is trivial and so is omitted.

Theorem 2

If , then problem (9) has exactly one solution.

Proof:

First, the energy functional is clearly nonnegative and proper because is a finite value. Let is a minimizing sequence of problem (9), i.e.,

Then, there is a constant so that

Thus,

(10)

Since , and , we have

By the Jensen’s inequality, we obtain

(11)

Namely, the sequences and are bounded in . Combining it with (10), we know that the sequences and are bounded in . Hence, there exists some such that, up to a subsequence,

(12)
(13)

By the (weak) lower semicontinuity of the and (13), we have

(14)

By (12) and the lower semicontinuity of , we have

(15)

Recalling the semicontinuity of norm, by (13) we have

(16)

Combining (14) and (15), we further have

Meanwhile, by (12) we have, up to a subsequence,

(17)

By and a.e., we obtain that satisfies the constraints and , a.e.. Thus is a solution of problem (9). The uniqueness of the solution is guaranteed by the strictly convex of the functional .

Iii The algorithm

Since there are two unknown variables in problem (9), we use the alternating minimization scheme to convert it into two subproblems. We describe it in Algorithm 1 in detail.

Algorithm 1.

  • Set initial value , , , predefined iterations and error tolerance .

  • At the th iteration:

    1. Given , compute by solving

      (18)
    2. Given , compute by solving

      (19)
  • Go back to step 2 until or .

Problems (18) and (19) are the constrained total variation minimal problems, which can be solved by the FGP algorithm [1] efficiently.

Suppose that the hazy image is an matrix. Let , and , define:

  • The unit ball in :

  • The divergence of :

  • The gradient of :

    where

  • The projection from on :

  • The projection from on :

Then, according to Proposition 4.1 in [1], the solution of problem (18) can be written as

where is the solution of the following problem:

(20)

with . Problem (20) is computed by

(21)

where is obtained by an accelerated scheme from :

(22)

Thus, the FGP algorithm for Problem (18) is as follows:

Algorithm 2.

  • Set initial values , and predefined iterations .

  • At the jth iteration:
    Given and , Compute

    1. by (21),

    2. and by (22).

  • , go back to step 2 until convergence or .

  • Set .

Since we set , we can solve problem (19) by the similar method.

Next, we investigate the convergence of Algorithm 1.

Theorem 3

Let and be the sequence derived from Algorithm 1. Then the sequence converges to some (up to a sequence), and for any

we have

Proof:

By (18) and (19), we have the following inequalities

Thus, is bounded and decreases with . Because of

(23)

we have

(24)

Since , and , we have

which implies that and are bounded in and so bounded in . By (24), we obtain that and are bounded in . Therefore, up to a subsequence, there is some so that

(25)
(26)

Since and decreases with , there exists such that

Now, we prove .

In fact, by the lower semicontinuity of the and , (25) and (26), we have

(27)

Therefore, we further have

(28)

On the other hand, by considering the inequalities

and

we have

(29)

The right-hand side of (29) can explicitly be written as

Since

by (29) we have

Letting , we have

which implies . Combining (28), we obtain .

Next, we show the second assertion.

In fact, for any , , we have

The above two inequalities imply that

Letting , we have

Similarly, for any , , we have

The proof is completed.

Iv Experimental results

In this section, we present some experimental results to show the performance and the effectiveness of our model, in comparison to other relevant models such as TH model [23], HST model [8], NN model [19] and GVPB model [7].

For the proposed model, we set , and ; and for Algorithm 1 and for Algorithm 2. For the other comparison models excluding NN model, we use the default parameters provided in their corresponding papers (or programs). For NN model, we use the airlight in the program downloaded from the authors’ homepage.

For color images, we process each of three color channels (R,G,B) separately and then synthesize the final results. For each color channel, the atmospheric light is theoretically estimated by:

where is color channel index, and is a positive constant. Since our model is insensitive to the value of (Figure 6 gives an example), is simply set as 255 in all experiments. The transmission is obtained by

where is derived from Algorithm 1. With the transmission and the atmospheric light , we can theoretically recover the each color channel of the scene radiance according to Equation (1); namely,

(30)

However, since the transmission may be very close to zero in some patches (e.g., sky regions), it is known from (30) that the directly recovered scene radiance will contain some black patches. As done in [8], we restrict the transmission to a lower bound . Therefore, the scene radiance is recovered by

(31)

Besides, since the scene radiance obtained by (31) looks a little dark, we perform a gamma correction with before recovering the final scene radiance. The value of can typically range from 0.1 to 0.4 by experiments. In the following experiments, we simply set .

In the following, we present some experiments of our model, in comparison to other four models. The original hazy images are shown in Figure 1 (a) to Figure 5 (a). Figure 1(b-e) to Figure 5 (b-e) show the results of the TH model [23], the HST model [8], the NN model [19] and the GVPB model [7], respectively. The results of our model are given in Figure 1(f) to Figure 6(f). We see from Figure 1(b) to Figure 5 (b) that the haze of the results obtained by the TH model [23] is removed clearly and the detail in the pictures is well preserved. However, the results significantly suffer from over-enhancement. This effect can be easily observed in Figure 2(b) and Figure 4(b). Moreover, halo artifacts appear near the discontinuities (see the intersection areas of the trees and the sky in Figure 4(b)). This is because the TH model tends to produce a dark sky. From Figure 1(c) to Figure 5 (c) we can see that the HST model [8] produces natural results and can dehaze image effectively. But the detail of the results is not well (see the grass ground in Figure 1(c) and the rock in Figure 4(c)). Because the refined transmission of the HST model contains too much detail, it can’t represent the depth information accurately. Besides, a halo effect can be observed in the sky in Figure 3(c). We can observe from Figure 1 (d) to Figure 5 (d) that NN model [19] is very effective for some images such as Figure 1 (d) and Figure 4(d). But it also suffers from over-enhancement for other images such as Figure 2 (d) Figure 4(d). Beside, we can see in Figure 5 (d), the result tends to be yellow. The reason why some of the results are good and others are not very satisfactory is that we set the parameters uniformly and the NN model is sensitive to the values of the parameters. We observe from Figure 1(e) to Figure 5(e) that the GVPB model [7] produces more natural results. But since it dehazes image based on enhancement which don’t utilize the depth information, it can’t clearly dehaze the far away region of the image whose scene depth varies from far to near. This effect can be easily observed in Figure 1(e) and Figure 4(e). Compared with the results of the four models, our results are free from halo effect and the details of our results are enhanced moderately, as displayed in Figure 1(f) to Figure 5(f). The sky in Figure 2(f) and Figure 3(f) is brighter and clear and the details in Figure 1(f) and Figure 4(f) are enhanced well. From Figure 5, we can see the result obtained by our method in Figure 5 has the best visual effect.

In the next experiment, we test the sensitivity of the parameters , , and . First, we fix , , and set and 30. From Figure 6(b)-Figure 6(e), we can observe that the change of the value of has almost no influence on the final results. Next, we fix , and set . From Figure 7(b)-Figure 7(d), we can see that there is almost no change in the enhanced images when the parameter varies from 10 to 1000. Last, we fix , and set . We can see from Figure 8(b)-Figure 8(d) that the results changes a little dark as the parameter varies from 0.01 to 1.

In this set of experiments, we evaluate the four comparison methods and our method quantitatively. For this purpose, we need the ground truth images. Therefore, we download the fog-free images from the dataset [31] and use the graduated fog filter of the Photoshop to synthesize 48 hazy images. Figure 9 shows some of the results. We use the total mean square error (MSE) index to evaluate these results objectively. The MSE is defined as:

where and are the obtained result and the ground truth, respectively, the subscript is color channel index, and is the number of the image pixels. From the definition, we know that the smaller the value of the MSE is, the better the result is. Table 1 lists the MSE values of the 48 images, from which we can see our method has the smallest MSE value averagely.

In the next experiment, we give a example to demonstrate the importance of the constraints in our model. Figure 10(a)-Figure 10 (c) are the hazy image and the dehazed images by our model without or with constraint, respectively. We can observe from Figure 10(b) that the model without the constraint can’t dehaze the image at all.

Since our method tackles the image channel separately, it also works for the gray-scale images. Figure 11 shows an example.

V Conclusion and Discussion

In this paper, we have introduced a new variational method for single image dehazing. The proposed method converts the problem of estimating the transmission map to a constrained variational problem. The existence and uniqueness of the solution of the proposed model are discussed. To solve the model efficiently, we adopt an alternating minimization scheme and the FGP algorithm. At last, we present some experimental results to test the effectiveness of our model.

It is still a challenging problem to dehaze image, especially for the images with both fine details and sky. Our method is to some extend effective for these images, but it still needs to further improve. For example, we simply choose in our method. This can make the results avoid the halo effect, but it is at the cost of losing some contrast. In the future, we will consider how to choose adaptively.

References

  • [1] A. Beck and M. Teboulle, Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems, IEEE Trans. Image Process., 18(2009), pp. 2419-2434.
  • [2] B. Chen and S. Huang, An advanced visibility restoration algorithm for single hazy images, ACM Trans. Multimed. Comput. Commun. Appl., 11(2015).
  • [3] L. K. Choi, J. You, and A. C. Bovik, Referenceless prediction of perceptual fog density and perceptual image defogging, IEEE Trans. Image Process., 24(2015), pp. 3888-3901.
  • [4] R. Fattal, Single image dehazing, In SIGGRAPH, 2008, pp. 1-9.
  • [5] F. Fang, F. Li, and T. Zeng, Single image dehazing and denoising: A fast variational approach, SIAM J. Imaging Sci., 7 (2014), pp. 969-996.
  • [6] R. FATTAL, Dehazing Using Color-Lines, ACM Trans. Graph., 34(2014), pp. 13:1-13:14
  • [7] A. Galdran, J. Vazquez-Corral, D. Pardo, and M. Bertalm, Enhanced variational image dehazing, SIAM J. Imaging Sci., 8(2015), pp. 1519-1546.
  • [8] K. He, J. Sun, and X. Tang, Single image haze removal using dark channel prior, IEEE Trans. Pattern Anal. Mach. Intell., 33 (2011), pp. 2341-2353.
  • [9] K. He, J. Sun, and X. Tang, Guided image filtering, IEEE Trans. Pattern Anal. Mach. Intell., 35 (2013), pp. 1397-1409.
  • [10] S. Huang, J. Ye, and B. Chen, An advanced single-image visibility restoration algorithm for real-world hazy scenes, IEEE Trans. Ind. Electron, 62(2015), pp. 2962-2972.
  • [11] R. KIMMEL, A variational framework for Retinex, Int. J. Comput. Vis., 52(2003), pp. 7-23.
  • [12] J. Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, and D. Lischinski, Deep photo: Model-based photograph enhancement and viewing, ACM Trans. Graph., 27(2008).
  • [13] J. Long, Z. Shi, W. Tang, and C. Zhang, Single remote sensing image dehazing, IEEE Geosci. Remote Sensing Lett., 11 (2014), pp. 59-63.
  • [14] Y. Lai, Y. Chen, C. Chiou, and C. Hsu, Single-image dehazing via optimal transmission map under scene priors, IEEE Trans. Circuits Syst. Video Technol., 25(2015), pp. 1-14.
  • [15] S. G. Narasimhan and S. K. Nayar, Vision and the atmosphere, Int. J. Comput. Vis., 48(2002), pp. 233-254.
  • [16] S. G. Narasimhan and S. K. Nayar, Interactive deweathering of an image using physical models, In Workshop on Color and Photometric Methods in Computer Vision, 2003.
  • [17] S. Narasimhan and S. Nayar, Contrast restoration of weather degraded images, IEEE Trans. Pattern Anal. Mach. Intell., 25 (2003), pp. 713-724.
  • [18] M. K. Ng and W. Wang, A total variation model for Retinex, SIAM J. Imaging Sci., 4 (2011), pp. 345-365.
  • [19] K. Nishino, L. Kratz and S. Lombardi, Bayesian defogging, Int J Comput Vis, 98 (2012) pp. 263-278.
  • [20] Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, Instant dehazing of images using polarization, CVPR, 1(2001), pp. 325-332.
  • [21] S. Shwartz, E. Namer, and Y. Y. Schechner, Blind haze separation, CVPR, 2(2006), pp. 1984 - 1991.
  • [22] R. Tan, Visibility in bad weather from a single image, CVPR, 2008, pp. 1-8.
  • [23] J.-P. Tarel and N. Hautière, Fast visibility restoration from a single color or gray level image, in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 2201-2208.
  • [24] Y. Wang and C. Fan, Single image defogging by multiscale depth fusion, IEEE Trans. Image Process., 23 (2014), pp. 4826-4837.
  • [25] W. Wang and C. He, A variational model with barrier functionals for Retinex, SIAM J. Imaging Sci., 8 (2015), pp. 1955-1980.
  • [26] J. Wang, N. He, L. Zhang, and K. Lu, Single image dehazing with a physical model and dark channel prior, Neurocomputing, 149 (2015), pp. 718-728.
  • [27] I. Yoon, S. Kim, D. Kim, M.H. Hayes, and J. Paik, Adaptive defogging with color correction in the HSV color space for consumer surveillance system,IEEE Trans. Consum. Electron., 58 (2012), pp. 111-116.
  • [28] Q. Zhu, J. Mai, and L. Shao, A fast single image haze removal algorithm using color attenuation prior, IEEE Trans. Image Process., 24(2015), pp. 3522-3533.
  • [29] Middleton, Vision through the atmosphere, University of Toronto Press, 1952.
  • [30] SK. Dai, JP, Tarel, Adaptive sky detection and preservation in dehazing algorithm, Proc. IEEE International Symposium on Intelligent Signal Processing and Communication Systems, 2015.
  • [31] Natural fog-free, foggy, and test images used in FADE:, http://live.ece.utexas.edu/research/fog/fade_defade.html.