STAR: A Structure and Texture Aware Retinex Model

06/16/2019 ∙ by Jun Xu, et al. ∙ 0

Retinex theory is developed mainly to decompose an image into the illumination and reflectance components by analyzing local image derivatives. In this theory, larger derivatives are attributed to the changes in piece-wise constant reflectance, while smaller derivatives are emerged in the smooth illumination. In this paper, we propose to utilize the exponentiated derivatives (with an exponent γ) of an observed image to generate a structure map when being amplified with γ>1 and a texture map when being shrank with γ<1. To this end, we design exponential filters for the local derivatives, and present their capability on extracting accurate structure and texture maps, influenced by the choices of exponents γ on the local derivatives. The extracted structure and texture maps are employed to regularize the illumination and reflectance components in Retinex decomposition. A novel Structure and Texture Aware Retinex (STAR) model is further proposed for illumination and reflectance decomposition of a single image. We solve the STAR model in an alternating minimization manner. Each sub-problem is transformed into a vectorized least squares regression with closed-form solution. Comprehensive experiments demonstrate that, the proposed STAR model produce better quantitative and qualitative performance than previous competing methods, on illumination and reflectance estimation, low-light image enhancement, and color correction. The code will be publicly released.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 3

page 6

page 7

page 8

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

(a) Input (d) Structure/Texture (b) Illumination (e) Enhancement (c) Reflectance (f) Color Correction
Fig. 1: An example to illustrate the applications of the proposed STAR model based on Retinex theory. (a) The input low-light/color-distorted image; (b) The estimated illumination component of (a); (c) The estimated reflectance component of (a); (d) The extracted structure and texture maps (half each) of (a); (e) The illumination enhanced low-light image of (a); (f) The color corrected image of (a).

The Retinex theory proposed by Land and McCann [1, 2] models the color perception of human vision on natural scenes. It can be viewed as a fundamental theory for intrinsic image decomposition problem [3], which aims to decomposing an image into reflectance and illumination (or shading). A simplified Retinex model involves decomposing an observed image into an illumination component and a reflectance component via , where denotes the element-wise multiplication. In the observed scene , the illumination expresses the color of the light striking the surfaces of objects, while the reflectance reflects the painted color of the surfaces of objects [4]

. Retinex theory has been applied in many computer vision tasks, such as image enhancement 

[4, 5, 6] and image/color correction [7, 8] (please refer to Figure 1 for an example).

The Retinex theory introduces a useful property of derivatives [1, 2, 4]

: larger derivatives are often attributed to the changes in reflectance, while smaller derivatives are from the smooth illumination. With this property, the Retinex decomposition can be performed by classifying the image gradients into the reflectance component and the illumination one 

[9]. However, binary classification of image gradient is unreliable since reflectance and illumination changes will coincide in an intermediate region [4]. Later, several methods are proposed to classify the edges or edge junctions, instead of gradient, according to some trained classifiers [10, 11]. However, it is quite challenging to train classifiers considering all possible ranges of reflectance and illumination configurations. Besides, though these methods explicitly utilize the property of derivatives, they perform Retinex decomposition by analyzing the gradients of a scene in a local manner, while ignoring the global consistency of the structure in the scene. To solve this drawback, several methods [4, 5, 6, 7, 8] perform global decomposition with the consideration of different regularizations. However, these methods ignore the property of derivatives and cannot separate well relectance and illumination.

In this paper, we introduce novel exponentiated local derivatives to better exploit the property of derivatives in a global manner. The exponentiated derivatives are determined by an introduced exponents on local derivatives, and generalize the trivial derivatives to extract structure and texture maps. Given an observed scene (e.g., Figure 1 (a)), its derivatives are exponentiated by to generate a structure map (Figure 1 (d) up) when being amplified with and a texture map (Figure 1 (d) down) when being shrank with . The extracted structure and texture maps are employed to regularize the illumination (Figure 1 (b)) and reflectance (Figure 1 (c)) components in Retinex decomposition, respectively. With the accurate structure and texture maps, we propose a Structure and Texture Aware Retinex (STAR) model to accurately estimate the illumination and reflectance components. We solve the STAR model in an alternating minimization manner. Each sub-problem is transformed into a vectorized least squares regression with closed-form solution. Comprehensive experiments demonstrate that, the proposed STAR model produces better quantitative and qualitative performance than previous competing methods, on illumination and reflectance estimation, low-light image enhancement, and color correction. In summary, the contribution of this work are three-fold:

  • We propose to employ exponentiated local derivatives to better extract the structure and texture maps.

  • We propose a novel Structure and Texture Aware Retinex (STAR) model to accurately estimate the illumination and reflectance components, and exploit the property of derivatives in a global manner.

  • Experimental results show that the proposed STAR model produces better quantitative and qualitative performance than previous competing methods on Retinex decomposition, low-light image enhancement, and color correction.

The remaining paper is organized as follows. In §II, we review the related work in this work. In §III, we introduce the proposed structure and texture awareness based weighting scheme. The proposed structure and texture aware Retinex model is proposed in §IV. §V describes the detailed experiments on Retinex decomposition of illumination and reflectance. §VI describes the proposed STAR model to two other image processing applications: low-light image enhancement and color correction. We conclude this paper in §VII.

(a) Input (b) (c) (d) (e)
Fig. 2:

Comparisons on the exponentiated total variation (ETV) and exponentiated mean local variance (EMLV) filters on structure and texture extraction. For an input RGB image (a), V refer to its Value channel in HSV space.

Ii Related Work

Ii-a Retinex model

The Retinex model has been extensively studied in literature [12, 13, 14, 15, 16, 17, 18, 19, 9, 10, 11, 20, 21, 22, 23, 24, 25, 26, 4, 5, 6, 27, 28, 29, 30, 7, 8, 31], which can be roughly divided into classical ones [12, 13, 14, 15, 16, 17, 18, 19] and variational ones [10, 11, 9, 20, 21, 22, 24, 25, 23, 26, 4, 5, 6, 27, 28, 29, 30, 7, 8, 31].

Classical Retinex Methods include path-based methods [12, 13, 14, 15]

, Partial Differential Equation (PDE)-based methods 

[16, 17], center/surround methods [18, 19]. Early path-based methods [12, 13] are developed based on the assumption that, the reflectance component can be computed by the product of ratios along some random paths. These methods demand careful parameter tuning and incur high computational costs. To improve the efficiency, later path-based methods of [14, 15] employ recursive matrix computation techniques to replace previous random path computation. However, their performance is largely influenced by the number of recursive iterations, and unstable for real applications. PDE-based algorithms [16, 17]

employ partial differential equation (PDE) to estimate the reflectance component, and can be solved efficiently by the fast Fourier transform (FFT). However, the structure of the illumination component will be degraded, since gradients derived by a divergence-free vector field often loss the expected piece-wise smoothness. The center/surround methods include the famous single-scale Retinex (SSR) 

[18] and multi-scale Retinex with color restoration (MSRCR) [19]. These methods often restrict the illumination component to be smooth, and the reflectance component to be non-smooth. However, due to lack of a structure-preserving restriction, SSR/MSRCR tend to generate halo artifacts around edges.

Variational Methods [10, 11, 9, 20, 21, 22, 24, 25, 23, 26, 4, 5, 6, 27, 28, 29, 30, 7, 8, 31] have been proposed for Retinex based illumination and reflectance estimation. In [9], the smooth assumption is introduced into a variational model to estimate the illumination. But this method is slow and ignores to regularize the reflectance. Later, an variational model is proposed in [24] to focus on estimating the reflectance. But this method ignores to regularize the illumination. The logarithmic transformation is also employed in [20] as a pre-processing step to suppresses the variation of gradient magnitude in bright regions, but the reflectance estimated with logarithmic regularizations tends to be over-smoothed. To consider both reflectance and illumination regularizations, a total variation (TV) model based method is proposed in [23]. But similar to [20], the reflectance is over-smoothed due to the side-effect of the logarithmic transformation. Recently, Fu et al. [32] developed a probabilistic method for simultaneous illumination and reflectance estimation (SIRE) in linear space. This method can preserve well the details and avoid over-smoothness of reflectance compared to the previous methods performing in the logarithmic space. To alleviate the detail loss problem of the reflectance component in logarithmic space, Fu et al. [7] proposed a weighted variational model (WVM) to enhance the variation of gradient magnitude in bright regions. However, the illumination may instead be damaged by the unconstrained isotropic smoothness assumption. By considering the properties of 3D objects, Cai et al. [8] proposed a Joint intrinsic-extrinsic Prior (JieP) model for Retinex decomposition. However, this model is prone to over-smoothing both the illumination and reflectance of a scene. In [31], Li et al. proposed the robust Retinex model considering an additional noise map, but this work is proposed only for low-light images accompanied by intensive noise.

Ii-B Intrinsic Image Decomposition

The Retinex model is in similar spirit with the intrinsic image decomposition model [33, 34, 35, 36, 37, 38, 39], which decomposes an observed image into Lambertian shading and reflectance (ignoring the specularity). The major goal of intrinsic image decomposition is to recover the shading and relectance terms from an observed scene, while the specularity term can be ignored without performance degradation [33]. However, the reflectance recovered in this problem usually loses the visual content of the scene [6], and hence can hardly be used for simultaneous illumination and reflectance estimation. Therefore, intrinsic image decomposition does not satisfy the purpose of Retinex decomposition for low-light image enhancement, in which the objective is to preserve the visual contents of dark regions as well as keep its visual realism [6]. For more difference between Retinex decomposition and intrinsic image decomposition, please refer to [6].

Iii Structure and Texture Awareness

In this section, we first present the simplified Retinex model, and then introduce structure and texture awareness for illumination and reflectance regularization.

Iii-a Simplified Retinex Model

The Retinex model [2] is a color perception model of the human vision system. Its physical goal is to decompose an observed image into its illumination and reflectance components, i.e.,

(1)

where means the illumination map of the scene representing the brightness of objects, denotes the surface reflection of the scene representing its physical characteristics, and means element-wise multiplication. The illumination and reflectance can be recovered by alternatively estimating them via

(2)

where means element-wise division. In fact, we employ and to avoid zero denominators, where .

To solve this inverse problem (2), previous methods usually employ an objective function that estimates illumination and reflectance components by

(3)

where and are two different regularization functions for illumination and reflectance , respectively. One implementation choice of and is the total variation (TV) [40], which is widely used in previous methods [23, 7].

Iii-B Structure and Texture Estimator

The Retinex model (1) decomposes an observed scene into its illumination and reflectance components. This problem is highly ill-posed, and proper priors of illumination and reflectance should be considered to regularize the solution space. Qualitatively speaking, the illumination should be piece-wisely smooth, capturing the structure of the objects in the scene, while reflectance should present the physical characteristics of the observed scene, capturing its texture information. Here, texture refers to the patterns in object surface, which are similar in local statistics [41].

Previous structure-texture decomposition methods often enforce the TV regularizers to preserve edges [23, 42, 28]. These TV regularizers simply enforce gradient similarity of the scene and extract the structure of the objects. There are two ways for structure-texture decomposition. One is to directly derive structure using structure-preserving techniques, such as edge-aware filters [43, 44] and optimization based methods [8]. The other way is to extract structure from the estimated texture weights [42]. However, these techniques [43, 42, 44, 8] are vulnerable to textures and produce ringing effect near edges. Moreover, the method [42] cannot extract scenes structures, whose appearances are similar to those of the underlying textures.

To better understand the power of these techniques for structure/texture extraction, we study two typical kinds of filters. The first is the TV filter [40], which computes the gradients of an input image as a guidance map:

(4)

The second one is the mean local variance (MLV) [8], which can also be utilized for structure map estimation:

(5)

where is the local patch around each pixel of , and its size is set as in all our experiments.

To support our idea that these TV and MLV filters can capture the structure of the scene, we visualize the effect of the two filters performed on extracting the structure/texture from an observed image. Here, the input RGB image Figure 2 (a), up) is first transformed into the Hue-Satuation-Value (HSV) domain. Since the Value (V) channel (Figure 2 (a) down) reflects the illumination and reflectance information, we process this channel for the input image. It can be seen from Figure 2 (c) that, the TV and MLV filters can basicly reflect the main structures of the input image. This point can be further validated by comparing the similarity of the two filtered image (Figure 2 (c)) with the edge extracted image of Figure 2 (a). To this end, we resort to a recently published edge detection method [45] to extract the main structure of the input image. By comparing the TV filtered image (Figure 3 (b)), MLV filtered image (Figure 3 (d)), and the edge extracted image (Figure 3 (c)), one can see that the TV and MLV filtered images already reflect the structure of the input image.

(a) Input (c) Edge detection of (a) (b) TV filtered (a) (d) MLV filtered (a)
Fig. 3: Comparison of TV filtered image (b), MLV filtered image (d), and the edge extracted image (c) of the input image (a). It can be seen that the TV and MLV filtered images can roughly reflect the structure of the input image.

Iii-C Proposed Structure and Texture Awareness

Existing TV and MLV filters described in Eqns. (4) and (5) cannot be directly employed in our problem, since they are prone to capture structural information. As described in Retinex theory, larger derivatives are attributed to the changes in reflectance, while smaller derivatives are emerged in the smooth illumination. Therefore, by exponential growth or decay, these local derivatives will . Here, we introduce an exponential version of them for flexible structure and texture estimation. Specifically, we add an exponent to the TV and MLV filters. In this way, we can make the two filters more flexible for separate structure and texture extraction. To this end, we propose the exponentiated TV (ETV) filter as

(6)

and the exponentiated MLV (EMLV) filter as

(7)

where is the exponent determining the sensitivity to the gradients of . Note that we evaluate the two exponentiated filters Eqns. (6) and (7) by visualizing their effects on a test image (i.e., Figure 2 (a), top). This RGB image is first transformed into the Hue-Satuation-Value (HSV) domain, and the decomposition is performed in the V channel. In Figure 2 (b)-(e), we plot the filtered images for the V channel of the input image. It is noteworthy that, with , the ETV and EMLV filters will reveal the textures of the test image, while with , the ETV and EMLV filters tend to extract the structural edges.

Motivated by this observation, we introduce a structure and texture aware scheme for illumination and reflectance decomposition. Specifically, we set , the ETV based weighting matrix as

(8)

and the EMLV based weighting matrix as:

(9)

where and are two exponential parameters to adjust the structure and texture awareness for illumination and reflectance decomposition. As will be demonstrated in §V, the values of and influence the effect of the Retinex decomposition performance. Due to considering local variance information, the EMLV filter (Eqn. 9) can reveal details and preserve structures better than the ETV filter (Figure 2). This point will also be validated in §V.

Iv Structure and Texture Aware Retinex Model

Iv-a Proposed Model

In this section, we propose a Structure and Texture Aware Retinex (STAR) model to estimate the illumination and the reflectance of an observed image , simultaneously. To make the STAR model as simple as possible, we adopt the TV -norm to regularize the illumination and reflectance components. The proposed STAR model is formulated as

(10)

where and are the two matrices defined in (9), indicating the structure map of the illumination and the texture map of the reflectance, respectively. The structure should be small enough to preserve the edges of objects in the scene, while large enough to suppress the details (as the inverse of Figure 2 (d,e)). The texture map should be small enough to reveal the details (as the inverse of Figure 2 (b,c)).

 

Algorithm 1: Solve the STAR Model (10)
Input: observed image , parameters ;
Initialization: , , set by (9);
for (do
1.  Update by Eqn. (13);
2.  Update by Eqn. (16);
  if (Converged)
3.   Stop;
  end if
end for
Output: Illuminance and Reflectance .

 

Iv-B Optimization Algorithm

Since the objective function is separable w.r.t. the two variables and , the problem (10) can be solved via an alternating direction method of multipliers (ADMM) algorithm [46]. The two separated sub-problems are convex and alternatively solved. We initialize the matrix variables . Denote and as the illuminance and reflectance variables at the -th () iteration, respectively, and is the iteration number. By optimizing one variable at a time while fixing the other, we can alternatively update the two variables as follows.

Update while fixing :

With in the -th iteration, the optimization problem with respect to becomes:

(11)

To solve the problem (11), we reformulate it into a vectorized format. To this end, with the vectorization operator , we denote vectors , , , , which are of length . Denote by the Toeplitz matrix from the discrete gradient operator with forward difference, then we have . Denote by , the matrices with lying on the main diagonals, respectively. Then, the problem (11) is transformed into a standard least squares regression problem:

(12)

By differentiating problem (12) with respect to , and setting the derivative to , we have the following solution

(13)

We then reformulate the obtained into matrix format via the inverse vectorization .

Update while fixing :

After acquiring from the above solution, the optimization problem (10) with respect to is similar to that of :

(14)

Similarly, we reformulate the problem (14) into a vectorized format. Additionally, we denote , . which are of length . We also have . Denote by , the matrices with lying on the main diagonals, respectively. Then, the problem (14) is also transformed into a standard least squares problem:

(15)

By differentiating problem (15) with respect to , and setting the derivative to , we have the following solution

(16)

We then reformulate the obtained into matrix format via inverse vectorization .

The above alternative updating steps are repeated until the convergence condition is satisfied or the number of iterations exceeds a preset threshold. The convergence condition of the ADMM algorithm is: and are simultaneously satisfied, or the maximum iteration number is achieved. We set and in our experiments. Since there are only two variables in problem (10), and each sub-problem has closed-form solution, it can be efficiently solved with convergence.

 

Algorithm 2: Alternative Updating Scheme
Input: observed image , parameters ;
Initialization: estimated by Algorithm 1;
for (do
1.  Update ;
2.  Update ;
3.  Solve the STAR model (10) and obtain and
   by Algorithm 1;
  if (Converged)
4.    Stop;
  end if
end for
Output: Final Illuminance and Reflectance .

 

Iv-C Updating Structure and Texture Awareness

Until now, we have obtained the decomposition of . To achieve better estimation on illumination and reflectance, we update the structure and texture aware maps and , and then solve the renewed problem (10). The alternative updating of (, ) and (, ) are repeated for iterations. The convergence condition of for this algorithm is: and are simultaneously satisfied, or the maximum updating iteration number is achieved. We set to balance the speed-accuracy tradeoff of the proposed STAR model in our experiments. We summarize the updating procedures in Algorithm 2.

V Experiments

In this section, we evaluate the qualitative and quantitative performance of the proposed Structure and Texture Aware Retinex (STAR) model, on Retinex decomposition (§V-B). In §V-C, we also perform an ablation study on illumination and reflectance decomposition to gain deeper insights into the proposed STAR Retinex model.

V-a Implementation Details

The input RGB-color image is first transformed into the Hue-Satuation-Value (HSV) space. Since the Value (V) channel reflects the illumination and reflectance information, we only process this channel, and transform the processed image from the HSV space to RGB-color space, similar to [7, 8]. In our experiments, we empirically set the parameters as . Due to considering local variance information, the EMLV filter (Eqn. 9) can reveal details and preserve structures better than the ETV filter (Figure 2). We will perform ablation study on this point in §V-C.

(a) Input (b) by WVM [7] (c) by JieP [8] (d) by STAR (e) by WVM [7] (f) by JieP [8] (g) by STAR
Fig. 4: Comparisons of illuminance and reflectance components by different Retinex decomposition methods.  and indicate illuminance and reflectance, respectively.

V-B Retinex Decomposition

The Retinex decomposition includes illumination and reflectance estimation. Accurate illumination estimation should not distort the structure, while being spatially smooth. Meanwhile, accurate reflectance should reveal the details of the observed scene. The ground truth for the illumination and reflectance is difficult to generate, and hence quantitative evaluation of the estimation is problematic.

To evaluate the effectiveness of the proposed STAR model, we perform qualitative comparisons on both illumination and reflectance estimation with two state-of-the-art Retinex models, including the Weighted Variation Model (WVM) [7], and the Joint intrinsic-extrinsic Prior (JieP) model [8]. Similar to these methods, we perform Retinex decomposition on the V channel of the HSV space, and transform the decomposed image back to the RGB space. Some visual results on several common test images are shown in Figure 4. It can be observed that, for the proposed STAR model, the structure awareness scheme enforces piece-wise smoothness, while the texture awareness scheme preserves details across the image. As can be seen in the Figures 4 (b)-(d) and (e)-(g), the proposed STAR method preserves better the structure of the three black regions on the white car, and reveals more details of the texture on the wall, than the other two methods of WVM [7] and JieP [8].

(a) Input (d) V channel of (a) (b) Illumination by STAR-ETV (e) Illumination by STAR-EMLV (c) Reflectance by STAR-ETV (f) Reflectance by STAR-EMLV
Fig. 5: Comparison of illumination and reflectance components by the proposed STAR model with ETV or EMLV weighting schemes. For better comparisons, we illustrate the components in the RGB channels instead of V channel.
(a) Input (e) V channel of (a) (b) Illumination by STAR w/o (f) Reflectance by STAR w/o (c) Illumination by STAR w/o (g) Reflectance by STAR w/o (d) Illumination by STAR (h) Reflectance by STAR
Fig. 6: Ablation study of structure or texture map on the illumination and reflectance decomposition performance of the proposed STAR model. For better comparisons, we illustrate the components in the RGB channels instead of V channel.
(a) Input (e) V channel of (a) (b) Illumination with (f) Reflectance with (c) Illumination with (g) Reflectance with (d) Illumination with (h) Reflectance with
Fig. 7: Ablation study of the parameters () on the illumination and reflectance decomposition performance of the proposed STAR model. For better comparisons, we illustrate the components in the RGB channels instead of V channel.
(a) Input (e) V channel of (a) (b) Illumination with (f) Reflectance with (c) Illumination with (g) Reflectance with (d) Illumination with (h) Reflectance with
Fig. 8: Ablation study of updating iterations on the illumination and reflectance decomposition performance of the proposed STAR model. For better comparisons, we illustrate the components in the RGB channels instead of V channel.

V-C Validation of the Proposed STAR Model

We conduct a detailed examination of our proposed STAR model for Retinex decomposition. We assess 1) the influence of the weighting scheme (ETV or EMLV) on STAR; 2) the importance of structure and texture awareness to STAR; 3) the influence of the parameters on STAR; 4) the necessity of updating structure and texture to STAR.

1. The influence of the weighting scheme (ETV or EMLV) on STAR. To study the the weighting scheme (ETV or EMLV) on STAR, we emply the ETV filter 8 and set and in (10) and update them as Algorithm 2 describes, and thus have another STAR model: STAR-ETV. The default STAR model can be termed as STAR-EMLV. From Figures 5, one can see that, the STAR-ETV model tends to provide little structure in illumination, while losing texture information in reflectance. By employing EMLV filter as the weighting matrix, the proposed STAR (STAR-EMLV) method maintains the structure/texture better than the STAR-ETV model.

2. Is structure and texture awareness important? To answer this question, we set or in (10) and update them as Algorithm 2 describes, and thus have two baselines: STAR w/o Structure and STAR w/o Texture. Note that if we set or in (10

) as comfortable identity matrix, the performance of the corresponding STAR model is very bad. From Figure 

6, one can see that, STAR w/o Structure tends to provide little structural information in illumination, while STAR w/o Texture influence little in illumination and reflectance. By considering both, the proposed STAR decompose the structure/texture components accurately.

3. How do the parameters and influence STAR? The are key parameters for the structure and texture awareness of STAR. In Figure 7, one can see that STAR with ((d) and (h)) produces reasonable results, STAR with ((c) and (g)) can barely distinguish the illumination and reflectance, while STAR with ((b) and (f)) confuses illumination and reflectance to a great extent. Since we regularize more on (), and in (f) are not exactly the same as and in (b), respectively.

4. Is Updating Necessary? We also study the effect of the updating iteration number on STAR. To do so, we simply set in STAR and evaluate its Retinex decomposition performance. From Figure 8, one can see that the illumination becomes more structual while reflectance reflects more details with more iterations.

Vi Other Applications

In this section, we apply the proposed STAR model on two other image processing applications: low-light image enhancement (§VI-A) and color correction (§VI-B).

Vi-a Low-light Image Enhancement

Capturing images in low-light environments suffers from unavoidable problems, such as low visibility and heavy noise degradation. Low-light image enhancement aims to alleviate this problem by improving the visibility and contrast of the observed images. To preserve the color information, the Retinex model based low-light image enhancement is often performed in the Value (V) channel of the Hue-Saturation-Value (HSV) domain.

Comparison Methods and Datasets. We compare the proposed STAR model with previous competing low-light image enhancement methods, including HE [47], MSRCR [19], Contextual and Variational Contrast (CVC) [48], Naturalness Preserved Enhancement (NPE) [5], LDR [49], SIRE [32], MF [50], WVM [7], LIME [6], and JieP [8]. We evaluate these methods on 35 images collected from [32, 50, 7, 6, 8], and on the 200 images in [5].

Objective Metrics. We qualitatively and quantitatively evaluate these methods on the subjective and objective quality metrics of enhanced images, respectively. The compared methods are evaluated on two commonly used metrics, one being the no-reference image quality assessment (IQA) metric Natural Image Quality Evaluator (NIQE) [51], and the other being the full-reference IAQ metric Visual Information Fidelity (VIF) [52]. A lower NIQE value indicates better image quality, and a higher VIF value indicates better visual quality. The reason we employ VIF is that it is widely considered to capture visual quality better than Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) [53], which cannot be used in this task since no “ground truth” images are available.

Results. We compare the proposed STAR model on the two sets of images previously mentioned. From Table I, one can see that the proposed STAR achieves lower NIQE and higher VIF results than the other competing methods. This indicates that the images enhanced by STAR present better visual quality than those of other methods. Besides, without the structure or texture weighting scheme, the proposed STAR model produces inferior performance on these two objective metrics. This demonstrates the effectiveness of the proposed structure and texture aware component for low-light image enhancement. In Figure 9, we compare the visual quality of state-of-the-art methods [5, 7, 6, 8]. As can be seen, on all the six cases, STAR achieves visually clear content while enhancing the illumination naturally, in agreement with our objective results. Besides, from the second row of Figure 9, one can observe that the proposed STAR also performs better than the competing methods [50, 7, 6, 8] on noise suppression.

(a) Input (b) NPE [5] (c) WVM [7] (d) LIME [6] (e) JieP [8] (f) STAR
Fig. 9: Comparisons of different Retinex methods on low-light image enhancement.

 

Dataset 35 Images 200 Images
Metric NIQE VIF NIQE VIF
Input 3.74 1.00 3.45 1.00
HE [47] 3.24 1.34 3.28 1.19
MSRCR [19] 2.98 1.84 3.21 1.11
CVC [48] 3.03 2.04 3.01 1.63
NPE [5] 3.10 2.48 3.12 1.62
LDR [49] 3.12 2.36 2.96 1.66
SIRE [32] 3.06 2.09 2.98 1.57
MF [50] 3.19 2.23 3.26 1.71
WVM [7] 2.98 2.22 2.99 1.68
LIME [6] 3.24 2.76 3.32 1.84
JieP [8] 3.06 2.67 3.18 1.82
STAR w/o 3.18 2.64 3.22 1.77
STAR w/o 3.09 2.78 3.01 1.82
STAR 2.93 2.96 2.86 1.92
TABLE I: Average NIQE [51] and VIF [52] results of different methods on 35 low-light images collected from [32, 50, 7, 6, 8] and 200 low-light images provided in [5].
(a) Input/Ground Truth (b) SIRE [32] (c) WVM [7] (d) JieP [8] (e) STAR
Fig. 10: Comparisons of different methods on color correction. Note that the proposed STAR method achieves more accurate correction performance when processing color-distorted images.

Vi-B Color Correction

In Retinex theory, if the estimation is performed in each channel of the RGB-color space, the estimated reflectance contains the original color information of the observed scene. Therefore, the Retinex model can be applied to color correction tasks. To demonstrate the estimation accuracy of the illumination and reflectance components, we evaluate the color correction performance of the proposed STAR model and the competing methods [54, 13, 55, 56, 57, 58, 59, 32, 7, 8].

We first compare the performance of the proposed STAR with three leading Retinex methods: SIRE [32], WVM [7] and JieP [8]. The original images and color corrected images are downloaded from the Color Constancy Website. In Figure 10, we provide some visual results of color correction using different methods. One can see that, the color of the wall and books in the 1st row, and the color of the orange bottle in the 3rd row. All these methods achieve satisfactory qualitative performance, when compared with the ground truth images in the 2nd and 4th rows of Figure 10 (a). To verify the accuracy of color correction using these methods, we employ the S-CIELAB color metric [60] to measure the color errors on spatial processing. The S-CIELAB errors between the ground truth and corrected images of different methods are shown in the 2nd and 4th rows of Figure 10 (b)-(e). As can be seen, the spatial locations of the errors, i.e., the green areas, of the STAR corrected images are much smaller than other methods. This indicates that the results of STAR are closer to the ground truth images (Figure 10 (a)) than other methods for color correction.

Furthermore, we perform a quantitative comparison of the proposed STAR with several leading color constancy methods [54, 13, 55, 56, 57, 58, 59, 8] on the Color-Checker dataset [57]. This dataset contains totally 568 images of indoor and outdoor scenes taken with two high quality DSLR cameras (Canon 5D and Canon1D). Each image contains a MacBeth color-checker for accuracy reference. The average illumination across each channel is computed in the RGB-color space separately, as the estimated illumination for that channel. The results in terms of Mean Angular Error (MAE, lower is better) between the corrected image and the ground truth image are listed in Table II. As can be seen, the proposed STAR method achieves lower MAE results than the competing methods on the color constancy problem.

 

Method White-Patch [13] Gray-Edge [56] Shades-Gray [55]
MAE 7.55 5.13 4.93
Method Bayesian [57] CNNs [58] Gray-World [54]
MAE 4.82 4.73 4.66
Method Gray-Pixel [59] JieP [8] STAR
MAE 4.60 4.32 4.11
TABLE II: Comparisons of color constancy with Mean Angular Errors (MAE) on the Color-Checker dataset [5].

Vii Conclusion

In this paper, we proposed a Structure and Texture Aware Retinex (STAR) model for illumination and reflectance decomposition. We first introduced an Exponentialized Mean Local Variance (EMLV) filter to extract the structure and texture maps from the observed image. The extracted maps were employed to regularize the illumination and reflectance components. In addition, we proposed to alternatively update the structure/texture maps, and estimate the illumination/reflectance for better Retinex decomposition performance. The proposed STAR model is efficiently solved by a standard ADMM algorithm. Comprehensive experiments on Retinex decomposition, low-light image enhancement, and color correction demonstrated that the proposed STAR model achieves better quantitative and qualitative performance than previous state-of-the-art Retinex decomposition methods.

References

  • [1] E. H. Land and J. J. McCann. Lightness and retinex theory. Josa, 61(1):1–11, 1971.
  • [2] E. H. Land. The Retinex theory of color vision. Scientific American, 237(6):108–129, 1977.
  • [3] Harry Barrow, J Tenenbaum, A Hanson, and E Riseman. Recovering intrinsic scene characteristics. Comput. Vis. Syst, 2:3–26, 1978.
  • [4] Q. Zhao, P. Tan, Q. Dai, L. Shen, E. Wu, and S. Lin. A closed-form solution to retinex with nonlocal texture constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1437–1444, 2012.
  • [5] S. Wang, J. Zheng, H. Hu, and B. Li. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, 22(9):3538–3548, 2013.
  • [6] X. Guo, Y. Li, and H. Ling. Lime: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 26(2):982–993, Feb 2017.
  • [7] X. Fu, D. Zeng, Y. Huang, X. Zhang, and X. Ding. A weighted variational model for simultaneous reflectance and illumination estimation. In CVPR, pages 2782–2790, 2016.
  • [8] B. Cai, X. Xu, K. Guo, K. Jia, B. Hu, and D. Tao. A joint intrinsic-extrinsic prior model for Retinex. In ICCV, pages 4000–4009, 2017.
  • [9] R. Kimmel, M. Elad, D. Shaked, R. Keshet, and I. Sobel. A variational framework for Retinex. International Journal of Computer Vision, 52(1):7–23, 2003.
  • [10] M. Bell and E. T. Freeman. Learning local evidence for shading and reflectance. In ICCV, volume 1, pages 670–677. IEEE, 2001.
  • [11] M. F. Tappen, W. T. Freeman, and E. H. Adelson. Recovering intrinsic images from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9):1459–1472, 2005.
  • [12] E. H. Land. Recent advances in Retinex theory and some implications for cortical computations: color vision and the natural image. Proceedings of the National Academy of Sciences, 80(16):5163–5169, 1983.
  • [13] D. H. Brainard and B. A. Wandell. Analysis of the Retinex theory of color vision. Journal of the Optical Society of America A, 3(10):1651–1661, 1986.
  • [14] J. A. Frankle and J. J. McCann. Method and apparatus for lightness imaging, 1983. US Patent 4,384,336.
  • [15] B. Funt, F. Ciurea, and J. McCann. Retinex in matlab. Journal of the Electronic Imaging, pages 48–57, 2004.
  • [16] B. K. P. Horn. Determining lightness from an image. Computer Graphics and Image Processing, 3(4):277 – 299, 1974.
  • [17] J. M. Morel, A. B. Petro, and C. Sbert. A pde formalization of Retinex theory. IEEE Transactions on Image Processing, 19(11):2825–2837, 2010.
  • [18] D. J. Jobson, Z. Rahman, and G. A. Woodell. Properties and performance of a center/surround Retinex. IEEE Transactions on Image Processing, 6(3):451–462, 1997.
  • [19] D. J. Jobson, Z. Rahman, and G. A. Woodell. A multiscale Retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing, 6(7):965–976, 1997.
  • [20] E. Provenzi, L. D. Carli, A. Rizzi, and D. Marini. Mathematical definition and analysis of the Retinex algorithm. Journal of the Optical Society of America A, 22(12):2613–2621, Dec 2005.
  • [21] M. Bertalmío, V. Caselles, and E. Provenzi. Issues about Retinex theory and contrast enhancement. International Journal of Computer Vision, 83(1):101–119, 2009.
  • [22] R. Palma-Amestoy, E. Provenzi, M. Bertalmío, and V. Caselles. A perceptually inspired variational framework for color enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(3):458–474, 2009.
  • [23] M. K. Ng and W. Wang. A total variation model for Retinex. SIAM Journal on Imaging Sciences, 4(1):345–365, 2011.
  • [24] W. Ma, J. M. Morel, S. Osher, and A. Chien. An -based variational model for Retinex theory and its application to medical images. In CVPR, pages 153–160. IEEE, 2011.
  • [25] W. Ma and S. Osher. A TV Bregman iterative model of Retinex theory. Inverse Problems and Imaging, 6(4):697–708, 2012.
  • [26] H. Li, L. Zhang, and H. Shen. A perceptually inspired variational method for the uneven intensity correction of remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 50(8):3053–3065, 2012.
  • [27] L. Wang, L. Xiao, H. Liu, and Z. Wei. Variational bayesian method for Retinex. IEEE Transactions on Image Processing, 23(8):3381–3396, 2014.
  • [28] J. Liang and X. Zhang. Retinex by higher order total variation decomposition. Journal of Mathematical Imaging and Vision, 52(3):345–355, 2015.
  • [29] J. M. Morel, A. B. Petro, and C. Sbert. A pde formalization of Retinex theory. IEEE Transactions on Image Processing, 19(11):2825–2837, 2010.
  • [30] D. Zosso, G. Tran, and S. Osher. Non-local Retinex—a unifying framework and beyond. SIAM Journal on Imaging Sciences, 8(2):787–826, 2015.
  • [31] M. Li, J. Liu, W. Yang, X. Sun, and Z. Guo. Structure-revealing low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing, 27(6):2828–2841, 2018.
  • [32] X. Fu, Y. Liao, D. Zeng, Y. Huang, X. Zhang, and X. Ding. A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation. IEEE Transactions on Image Processing, 24(12):4965–4977, Dec 2015.
  • [33] R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, pages 2335–2342, 2009.
  • [34] J. Shen, X. Yang, Y. Jia, and X. Li. Intrinsic images using optimization. In CVPR, 2011.
  • [35] J. T. Barron and Jitendra Malik. Color constancy, intrinsic images, and shape estimation. In ECCV, 2012.
  • [36] J. T. Barron and Jitendra Malik. Intrinsic scene properties from a single rgb-d image. In CVPR, 2013.
  • [37] Y. Li and M. S. Brown. Single image layer separation using relative smoothness. In CVPR, June 2014.
  • [38] J. T. Barron. Convolutional color constancy. In Proceedings of the IEEE International Conference on Computer Vision, pages 379–387, 2015.
  • [39] J. T. Barron and Y. Tsai. Fast fourier color constancy. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    , pages 886–894, 2017.
  • [40] L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992.
  • [41] L. Wei, S. Lefebvre, V. Kwatra, and G. Turk. State of the art in example-based texture synthesis. In Eurographics 2009, State of the Art Report. Eurographics Association, 2009.
  • [42] L. Xu, Q. Yan, Y. Xia, and J. Jia. Structure extraction from texture via relative total variation. ACM Transaction on Graphics, 31(6):139:1–139:10, 2012.
  • [43] R. Manduchi and C. Tomasi. Bilateral filtering for gray and color images. In ICCV, 1998.
  • [44] Q. Zhang, X. Shen, L. Xu, and J. Jia. Rolling guidance filter. In ECCV, pages 815–830, 2014.
  • [45] Y. Liu, M.-M. Cheng, X. Hu, J.g Bian, L. Zhang, X. Bai, and J. Tang. Richer convolutional features for edge detection. IEEE Trans. Pattern Anal. Mach. Intell., 2019.
  • [46] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers.

    Foundations and Trends® in Machine Learning

    , 3(1):1–122, 2011.
  • [47] H. Cheng and X. Shi. A simple and effective histogram equalization approach to image enhancement. Digital Signal Processing, 14(2):158–170, 2004.
  • [48] T. Celik and T. Tjahjadi. Contextual and variational contrast enhancement. IEEE Transactions on Image Processing, 20(12):3431–3441, 2011.
  • [49] C. Lee, C. Lee, and C. Kim. Contrast enhancement based on layered difference representation of 2d histograms. IEEE Transactions on Image Processing, 22(12):5372–5384, 2013.
  • [50] X. Fu, D. Zeng, Y. Huang, Y. Liao, X. Ding, and J. Paisley. A fusion-based enhancing method for weakly illuminated images. Signal Processing, 129:82–96, 2016.
  • [51] A. Mittal, R. Soundararajan, and A. C. Bovik. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3):209–212, March.
  • [52] H. R. Sheikh and A. C. Bovik. Image information and visual quality. IEEE Transactions on Image Processing, 15(2):430–444, Feb 2006.
  • [53] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  • [54] G. Buchsbaum. A spatial processor model for object colour perception. Journal of the Franklin Institute, 310(1):1–26, 1980.
  • [55] G. D. Finlayson and E. Trezzi. Shades of gray and colour constancy. In Color and Imaging Conference, volume 2004, pages 37–41, 2004.
  • [56] J. Van De Weijer, T. Gevers, and A. Gijsenij. Edge-based color constancy. IEEE Transactions on Image Processing, 16(9):2207–2214, 2007.
  • [57] P. V. Gehler, C. Rother, A. Blake, T. Minka, and T. Sharp. Bayesian color constancy revisited. In CVPR, pages 1–8. IEEE, 2008.
  • [58] S. Bianco, C. Cusano, and R. Schettini. Color constancy using CNNs. In CVPR Workshops, pages 81–89, 2015.
  • [59] K. Yang, S. Gao, and Y. Li. Efficient illuminant estimation for color constancy using grey pixels. In CVPR, pages 2254–2263, 2015.
  • [60] X. Zhang and B. A. Wandell. A spatial extension of cielab for digital color image reproduction. In SID International Symposium Digest of Technical Papers, volume 27, pages 731–734. Citeseer, 1996.