Exploring the significance of using perceptually relevant image decolorization method for scene classification

12/29/2017
by   V. Sowmya, et al.
0

A color image contains luminance and chrominance components representing the intensity and color information respectively. The objective of the work presented in this paper is to show the significance of incorporating the chrominance information for the task of scene classification. An improved color-to-grayscale image conversion algorithm by effectively incorporating the chrominance information is proposed using color-to-gay structure similarity index (C2G-SSIM) and singular value decomposition (SVD) to improve the perceptual quality of the converted grayscale images. The experimental result analysis based on the image quality assessment for image decolorization called C2G-SSIM and success rate (Cadik and COLOR250 datasets) shows that the proposed image decolorization technique performs better than 8 existing benchmark algorithms for image decolorization. In the second part of the paper, the effectiveness of incorporating the chrominance component in scene classification task is demonstrated using the deep belief network (DBN) based image classification system developed using dense scale invariant feature transform (SIFT) as features. The levels of chrominance information incorporated by the proposed image decolorization technique is confirmed by the improvement in the overall scene classification accuracy . Also, the overall scene classification performance is improved by the combination of models obtained using the proposed and the conventional decolorization methods.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

page 12

page 14

page 15

12/27/2018

No-Reference Color Image Quality Assessment: From Entropy to Perceptual Quality

This paper presents a high-performance general-purpose no-reference (NR)...
11/27/2013

Color and Shape Content Based Image Classification using RBF Network and PSO Technique: A Survey

The improvement of the accuracy of image query retrieval used image clas...
11/30/2017

A Color Intensity Invariant Low Level Feature Optimization Framework for Image Quality Assessment

Image Quality Assessment (IQA) algorithms evaluate the perceptual qualit...
07/09/2016

Combining multiple resolutions into hierarchical representations for kernel-based image classification

Geographic object-based image analysis (GEOBIA) framework has gained inc...
11/06/2020

Efficient Robust Watermarking Based on Quaternion Singular Value Decomposition and Coefficient Pair Selection

Quaternion singular value decomposition (QSVD) is a robust technique of ...
10/06/2020

Visualizing Color-wise Saliency of Black-Box Image Classification Models

Image classification based on machine learning is being commonly used. H...
02/17/2019

Semantically Interpretable and Controllable Filter Sets

In this paper, we generate and control semantically interpretable filter...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The two main modules of any image classification system are feature extraction and classification. The features are extracted from the input color image in RGB space by processing the three color planes independently. However, most of the existing image classification systems, in particular, scene classification systems converts the input color image to grayscale, (so called color-to-grayscale image conversion) prior to feature extraction

[16, 22, 10, 5, 7].

The color-to-grayscale image conversion is performed to reduce the computational complexity of the features extracted from the single gray plane rather than from the three dimensional color planes. Also, the incorporation of significant chrominance information along with the contrast details in the converted grayscale images are essential, as the features required to categorize or classify the scene information are extracted from the converted grayscale images. But, the major challenge in the color-to-grayscale image conversion is the preservation of luminance information along with the chrominance contrast in the converted grayscale images as in the original color image

[1].

For example, the pattern represented by the variation of color in the input color image (Fig.1 (c)) is missed, when converted to gray image (Fig.1 (d)) by the standard and most commonly used NTSC rule which combines the red, blue and green channel with the ratio 0.3: 0.6: 0.1 respectively [23]. Similarly, there is absence of sun and number information in the converted gray scale images (Fig.1 (b) & (f)). Therefore, many methods were proposed to convert the color image to grayscale with the preservation of luminance and color contrast information [11, 17, 9, 2, 3]. The description and the findings of the benchmark image decolorization techniques exist in the literature is given in Table 1.

Figure 1: Illustration of conventional color-to-grayscale image conversion. (a), (c) and (e) : Original Image. (b), (d) and (f) : Grayscale version obtained using standard NTSC rule

.

Authors Dataset Used Method Description Findings
CIE [14]
Color-to-gray
benchmark dataset
The luminance information ( Y ) of the CIE XYZ space is used
to achieve the grayscale version of the original color image
This method fails in the case of images with isoluminous colors
Grundland
and Dodgson [8]
Color photographs
The luminance information is added to the constant amount of
chrominance information to enhance the grayscale contrast
The performance of this method is limited to the
color discrimination present in the original image
Rasche et al. [17]
Color-to-gray
benchmark dataset
and natural images
A constrained multivariate optimization technique is used
to preserve the color contrast of the original image
Computational costs of the optimization technique results in
artifacts in the natural images with continuous tones
Smith et al. [18]
Color-to-gray
benchmark dataset
and natural images
A global mapping based on Helmholtz-Kohlrausch effect
along with the local chromatic contrast enhancement is
employed in this color-to-grayscale conversion technique
This technique relies on user control and also,
the visual appearance of constant color region is affected
Bala [14]
Color-to-gray
benchmark dataset
High-frequency chrominance information is embedded
into the luminance to preserve the chrominance edges locally
The method is not applicable to complex images
Neumann et al. [15]
Color-to-gray
benchmark dataset
Consistent color gradients are selected locally and 2D
fast integration is performed to attain the grayscale image
The details of the image may be lost by the unpredictable
behaviour in inconsistent regions of the gradient field
Gooch et al. [2]
Color-to-gray
benchmark dataset
An objective function based on the local contrast between
pixel pairs is minimized using optimization technique
to preserve the color contrast of the original image
Computationally expensive and the quality of the output image
depends on the user input parameters
Lu et al. [3]
Color-to-gray
benchmark dataset
and natural images
Bimodal energy function is used to attain a flexible contrast
preserving constraint
Complex optimization steps increases the cost of computation
Lu et al. [12]
Color-to-gray
benchmark dataset
and natural images
An effective and very fast optimization based approach
is used to maximally preserve the color contrast
This method fails in the case of color blindness test images
Sowmya et al. [20]
Color-to-gray
benchmark dataset
and natural images
An effective singular value decomposition (SVD) based
approach is used to incorporate the significant chrominance
information in the converted grayscale images.
This method fails to preserve the contrast due to the fixed value
of the weightage parameter
Table 1: Description of existing methods

Most of the existing methods for color-to-grayscale image conversion is based on optimization techniques which may be computationally complex [11, 17, 9, 2, 3]. Therefore, simple image decolorization technique using singular value decomposition (SVD) was proposed [20]. Recently, an algorithm called color-to-gray structure similarity (C2G-SSIM) is developed to measure the level of chrominance and structure information incorporated in the converted grayscale images [13]. In [13]

, one of the applications of C2G-SSIM is demonstrated as the parameter tuning of color-to-grayscale image conversion algorithms motivated to use the C2G-SSIM along with the SVD to develop an effective and perceptually improved method for image decolorization, which preserves the contrast information. SVD algorithm is used in our proposed work to reconstruct the chrominance planes by using the significant chrominance information captured through the singular values and the corresponding Eigenvectors.

Unlike the existing SVD based image decolorization method [20], the proposed framework avoids the constant weight of the chrominance information added to the luminance. Instead, the weightage of the chrominance information added to the luminance is computed using the C2G-SSIM. In [5]

, the performance of the scene classification is improved by modeling the dense scale invariant features using gaussian mixtures and adapted gaussian mixture models. Later, the deep features to improve the scene recognition task are learnt by convolutional neural networks (CNN) using color images

[6]

. The performance of the scene classification system is improved by the combination of deep belief networks and support vector machines trained using the grayscale images with better chrominance representations

[19]. Therefore, the objective of the present work is to verify the effectiveness of the chrominance information incorporated by the proposed image decolorization technique by applying the developed color-to-grayscale image conversion algorithm in the experiments for scene classification. Also, the performance of the scene classification system must be improved by the combination of the proposed image decolorization technique with the standard color-to-grayscale image conversion method.

The proposed novelties of the present work are as follows:

  • The new method for effective and perceptually improved image decolorization method based on SVD and C2G-SSIM is proposed.

  • The role of significant and relevant chrominance information with the preservation of contrast details incorporated by the proposed method is applied for scene classification.

  • A methodology is adopted for overall improvement in the scene classification performance by combining the models developed using the perceptually better grayscale images obtained from the proposed image decolorization technique with the most commonly used image decolorization method.

The organization of the paper is as follows: The basis for the present work is discussed in section 2. Section 3 presents the proposed method followed by experimental results and analysis in section 4. Section 5 contains the summary and conclusion of the present work.

2 Basis for the present work

The basis for the present work is discussed in this section.

The C2G-SSIM index is computed by the following three stages, namely (i) color space transformation, (ii) measurement of luminance (L), contrast (C) and structure (S) information & (iii) combination of the measured contents based on the type of input color image [13]. In the first stage, the input color image and the converted grayscale image are transformed to CIEL*a*b* color space. The second stage involves the computation of similarity measurement of luminance, contrast and structure information between the reference color image and the converted gray image in CIEL*a*b* space. The luminance plane (of the reference color image and the converted gray image) and the chrominance planes of the reference color image is processed as windows of uniform patch size with two-dimensional Gaussian filter centered at a pixel location for every window. The luminance measure is computed using Eq. (1) [13].

(1)

where is the mean luminance computed for the luminance plane of the reference color image in CIEL*a*b* color space at the pixel location . is the mean luminance computed for the luminance plane of the converted gray image in CIEL*a*b* color space at the same pixel location . C1 is a small positive stabilizing constant.

The chrominance measure is computed using the Eq. (2) [13].

(2)

where is the weighted mean color difference from its surroundings computed by considering the luminance and the chrominance details of the reference color image in CIEL*a*b* color space at the pixel location . is the mean gray tone difference computed at the same pixel location for the luminance plane of the converted gray image in CIEL*a*b* color space. C2 is a small positive stabilizing constant.

The structure measure is computed using the Eq. (3) [13].

(3)

where

is the standard deviation of the weighted mean color difference from its surroundings computed by considering the luminance and the chrominance details of the reference color image in CIEL*a*b* color space at the pixel location

. is the standard deviation of the mean gray tone difference computed at the same pixel location for the luminance plane of the converted gray image in CIEL*a*b* color space. is the cross correlation computed between and at the same pixel location . C3 is a small positive stabilizing constant.

Thus, the overall C2G-SSIM index is computed by combining all the three contents, namely luminance , contrast and structure at the pixel location using the Eq. (7) [13].

(4)

where , , . In [13], the expression for C2G-SSIM index is simplified by setting . And, for photographic images and for synthetic images.

3 Proposed Method

An image decolorization method is proposed to perceptually improve the performance of the color-to-grayscale image conversion using C2G-SSIM and SVD. The algorithm of the proposed image decolorization is discussed below. The block diagram of the proposed methodology shown in Fig.3 is explained below.

  1. Input Color Image: , where and refers to the row and column dimension of the input color image. The third dimension refers to the three color planes namely red, green and blue.

  2. Image Transformation: , where represents the transformed image in CIEL*a*b* color space which is the better suited color space for image editing [20]. is the color space transformation function which converts the input color image in RGB space to CIEL*a*b* space.

  3. Singular Value Decomposition: Each of the chrominance planes of the transformed image is subjected to singular value decomposition. , where refers to the first and the second chrominance planes of the transformed color image for respectively., and contains the Eigen vectors corresponding to the input chrominance planes. contains the Eigen values corresponding to the input chrominance planes.

  4. Chrominance Planes Reconstruction (): Each of the chrominance planes is reconstructed using the Eigen values and the corresponding Eigen vectors selected based on the rank of the input chrominance plane. .

  5. Gray/Decolorized Image(): The gray image is obtained using the equation (5)

    (5)
  6. Transformation to RGB color space & average computation: , where transforms the gray image to RGB space with only the luminance information. The average of three planes in RGB space (, , ) is computed using the equation (6) to form the final gray or decolorized image [20].

    (6)

    Though the output gray image obtained by the existing SVD image decolorization technique is comparable with all the benchmark algorithms, the weightage parameter of the chrominance information added to the luminance is fixed experimentally as 0.25 [20]. The fixed value of weightage parameter fails to preserve the contrast in the converted grayscale image which affects the perceptual quality of the converted image.

    Figure 2: Illustration of color-to-grayscale image conversion by SVD image decolorization method [20]. (a) : Original Image. Grayscale version obtained using (b) : SVD image decolorization (c) : Proposed Method

    .

    For example, the color contrast information present in the original color image (region where apple and orange in present) is not preserved in the grayscale image converted by the existing SVD based image decolorization (Fig.2(b)). Fig.2 (c) shows perceptually improved image decolorization from Fig.2 (a) as compared to the grayscale image in Fig.2 (b) which is obtained using the conventional SVD based method.

    The constant value is chosen using the measure designed to measure the significant components of color-to-grayscale image conversion such as luminance, structure and contrast information known as C2G-SSIM. The mathematical representation corresponding to our proposed method for color-to-grayscale image conversion is given by:

    (7)

    where is the C2G-SSIM index computed using the Eq. (7).

Figure 3: Block diagram of the proposed methodology

.

4 Experimental Results and Analysis

The proposed image decolorization technique to improve the perceptual quality of the image decolorization is evaluated as follows:

  1. Objective evaluation using quality assessment metric for image decolorization namely color-to-gray structure similarity index (C2G-SSIM) [13].

  2. Role of incorporating significant chrominance information by the proposed perceptually improved image decolorization method for scene classification.

4.1 Objective evaluation of the proposed perceptually improved SVD image decolorization using C2G-SSIM metric

The proposed method to improve the perceptual quality of the image decolorization technique using C2G-SSIM is experimented on two different benchmark datasets used for color-to-grayscale image conversion. The first dataset known as Cadik, contains 25 different color images which includes isoluminous (images with uniform light intensity values) color images [14]. The second dataset is COLOR250, which contains 250 different color images [4]. COLOR250 dataset contains generic color images along with the isoluminous pie chart color images. The proposed technique is experimented for 20 different values of the weight parameter () ranging from 0.05 to 1.0 with the incremental step value of 0.05.

The C2G-SSIM index is computed for the different values of for all the images in Cadik and COLOR250 datasets for which the samples are tabulated in Table 2. The experimental analysis computed for all the images in both the dataset shows that the best C2G-SSIM index is obtained for different value of other than as fixed in SVD decolorization algorithm for color-to-grayscale image conversion. For instance, the color images from Cadik dataset namely ‘balls0_color’ and ‘tulips’ have 0.9056 and 0.7860 respectively as C2G-SSIM index for , which is better than the index obtained as 0.8553 and 0.7057 repsectively for , as fixed in SVD decolorization method. Similar analysis are observed for all the images in both the datasets. For example, color image namely ‘im_8’ and ‘im_10’ in COLOR250 dataset have C2G-SSIM index as 0.8964 and 0.8090 for , which is better than the index obtained for the same images at . (SVD decolorization).

Image_name c-value
0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1.0
balls0_color 0.9056 0.8866 0.8553 0.8153 0.7664 0.7110 0.6557 0.6285 0.6014 0.5744 0.5478 0.5217 0.4963 0.4718
ColorsPastel 0.7314 0.7374 0.7519 0.7769 0.7858 0.8023 0.8170 0.8168 8143 0.8112 8076 0.8042 0.8008 0.7967
IM2-color 0.8094 0.8474 0.8883 0.9141 0.9305 0.9380 0.9396 0.9400 0.9398 0.9399 0.9395 0.9381 0.9365 0.9343
tulips 0.7860 0.7538 0.7057 0.6534 0.6144 0.5967 0.5709 0.5575 0.5444 0.5317 0.5194 0.5072 0.4954 0.4838
im_8 0.8838 0.8964 0.8692 0.8256 0.7880 0.7526 0.7285 0.7191 0.7115 0.7042 0.6976 0.6914 0.6852 0.6796
im_13 0.9103 0.9154 0.9171 0.9150 0.9079 0.8883 0.8641 0.8539 0.8445 0.8351 0.8254 0.8156 0.8057 0.7954
im_10 0.8048 0.8090 0.8075 0.8011 0.7873 0.7633 0.7245 0.7054 0.6879 0.6717 0.6563 0.6417 0.6279 0.6143
Table 2: C2G-SSIM index computed for different values of parameter (c) for the images in Cadik and COLOR250 dataset [14], [4]

The visual comparison of the gray image ‘Sunrise312’ obtained using the proposed technique and all the benchmark technique is shown in Fig.4. The gray image obtained using the proposed technique (Fig.4 (l)) has better contrast along with the structure content as in the original color image (Fig. 4 (a)). Hence, the C2G-SSIM index computed for the gray image converted using the proposed technique is better than all other existing benchmark color-to-gray image conversion techniques. The similar observation is also recorded for the pie chart image namely ‘im_203’ present in COLOR250 dataset. The variation in the colors present in the input color image (Fig.5 (a)) is missed in the corresponding gray images obtained using the existing color-to-gray conversion algorithms (Fig.5 (b), (d), (f) & (g)). Whereas, in the case of gray image obtained using the proposed technique (Fig.4 (i)), the color variation present in the input color image is represented with better contrast and structure in gray intensity level. Hence, the C2G-SSIM index computed for the gray image (im_203) obtained using the proposed technique is 0.9927 (Table 4), which is better than the index computed for all the existing benchmark algorithms. The reported trend of the proposed color-to-grayscale image conversion method holds good for all the sample images taken from COLOR250 dataset as tabulated in Table 4.

Figure 4: Visual Comparison of the proposed color-to-grayscale image conversion against the existing benchmark techniques for a sample image (Sunrise312) in Cadik dataset [14] (a): Input Color Image. (b)-(k): Grayscale version obtained using the existing benchmark methods namely rgb2gray, CIE_Y, Decolor, Rasche, Smith, Bala, Neuman, Gooch, RT-CP, SVD respectively. (l): Gray image obtained using the proposed technique

.

Figure 5: Visual Comparison of the proposed color-to-grayscale image conversion against the existing benchmark techniques for a sample image (im_203) in COLOR250 dataset [4] (a): Input Color Image. (b)-(h): Grayscale version obtained using the existing benchmark methods namely rgb2gray, RT-CP, CIE_Y, Decolor, Smith, Gooch, SVD respectively. (i): Gray image obtained using the proposed technique

.

The proposed color-to-grayscale image conversion based on the variation of the parameter () using C2G-SSIM index is compared against the existing benchmark techniques for all the images available in Cadik and COLOR250 datasets. The C2G-SSIM index computed for the sample images from both the datasets for the proposed method and existing benchmark techniques for color-to-grayscale image conversion are tabulated in Table 3 & 4. The C2G-SSIM value computed for the gray image obtained using the proposed decolorization algorithm is better than all the existing benchmark techniques. For example, in case of ‘Sunrise312’ (Cadik dataset) in Table 3, the C2G-SSIM index obtained for the gray image converted using the proposed technique is 0.8898 which is better than the values obtained for the grayscale image converted using all the benchmark algorithms.

Image_name rgb2gray [23] CIE_Y [14] Decolor [14] Rasche [14] Smith [14] Bala [14] Neuman [14] Gooch [14] CP [14] RT-CP [12] SVD [20] Proposed
155_5572 0.8554 0.8589 0.8424 0.6629 0.8584 0.8233 0.8212 0.8277 0.7749 0.8303 0.8147 0.8647
34445 0.8638 0.8595 0.8356 0.8556 0.8551 0.7681 0.7279 0.8592 0.8246 0.8540 0.8579 0.8659
portrait_4v 0.9289 0.9225 0.9269 0.9228 0.9270 0.8849 0.8901 0.9294 0.8836 0.9318 0.9234 0.9323
Sunrise312 0.8576 0.8455 0.8721 0.8247 0.8567 0.8337 0.8312 0.8688 0.7473 0.8810 0.8839 0.8898
watch 0.9611 0.9506 0.9519 0.9569 0.9587 0.8904 0.8112 0.9612 0.9277 0.9628 0.9611 0.9642
Table 3: Comparison of the proposed method for color-to-gray image conversion against the existing benchmark techniques for Cadik dataset based on C2G-SSIM quality index
Image_name rgb2gray [23] RT-CP [12] CP [14] CIE_Y [14] Decolor [14] Smith [14] Gooch [14] SVD [20] Proposed
im_18 0.8483 0.7815 0.8372 0.8474 0.8129 0.8469 0.8186 0.8332 0.8580
im_38 0.9061 0.8732 0.8802 0.9176 0.8749 0.9174 0.9111 0.8971 0.9236
im_66 0.9610 0.9625 0.9622 0.9642 0.9408 0.9651 0.9606 0.9634 0.9708
im_89 0.9229 0.9194 0.9187 0.9329 0.8746 0.9323 0.9256 0.9256 0.9334
im_117 0.9579 0.9497 0.9511 0.9553 0.9128 0.9544 0.9556 0.9502 0.9607
im_141 0.9685 0.9633 0.9649 0.9569 0.9562 0.9577 0.9626 0.9503 0.9707
im_160 0.7585 0.6263 0.6752 0.7596 0.7558 0.7451 0.7414 0.7160 0.7774
im_175 0.8934 0.8979 0.8891 0.8902 0.8656 0.8897 0.8729 0.8747 0.9001
im_187 0.8718 0.8585 0.8272 0.8851 0.7625 0.8742 0.8822 0.8850 0.8860
im_203 0.9868 0.9914 0.9895 0.9885 0.9909 0.9883 0.9865 0.9920 0.9927
im_227 0.9004 0.8601 0.8560 0.8966 0.9110 0.8945 0.8828 0.9243 0.9243
Table 4: Comparison of the proposed method for color-to-gray image conversion against the existing benchmark techniques for COLOR250 dataset based on C2G-SSIM quality index
Figure 6: Graphical representation of success rate of the proposed color-to-grayscale image conversion against the existing benchmark techniques for the images in Cadik dataset [14]

.

Figure 7: Graphical representation of success rate of the proposed color-to-grayscale image conversion against the existing benchmark techniques for the images in COLOR250 dataset [4]

.

Figure 8: Graphical representation of average C2G-SSIM index computed for the proposed color-to-grayscale image conversion and the existing benchmark techniques using all the images in Cadik dataset [14]

.

Figure 9: Graphical representation of average C2G-SSIM index computed for the proposed color-to-grayscale image conversion and the existing benchmark techniques using all the images in COLOR250 dataset [4]

.

The success rate of a technique is defined as the number of images whose C2G-SSIM score is maximum in the corresponding technique when compared with the other existing techniques [21]. The success rate computed for the proposed technique and all the existing algorithms for color-to-grayscale image conversion is represented graphically in Fig.6 and Fig.7 for Cadik and COLOR250 datasets respectively. Out of 25 images available in Cadik dataset, none of the images obtained maximum C2G-SSIM score for the gray images obtained using the existing algorithms namely Decolor, Bala, Neuman, Gooch. Therefore, the success rate for all these techniques is 0. The success rate for the proposed technique is 4 which is better than the success rate obtained for SVD decolorization, RT-CP, Rasche, CIE_Y and rgb2gray.

In case of COLOR250 dataset which contains 250 different color images, the success rate of the proposed technique is 113 which is greater than the success rate obtained for all other existing image decolorization algorithms. The success rate of the existing techniques namely CP and Smith is 7 in case of Cadik dataset, which is higher than that of the proposed technique (Fig.6). But, in case of COLOR250 dataset, the success rate of CP and Smith are 37 and 25 respectively, which is lower than that of the proposed technique (Fig.7).

The average C2G-SSIM index computed for all the images in both the dataset for the proposed technique and all the existing image decolorization algorithms are represented graphically in Fig.8 and Fig.9. The average C2G-SSIM index computed for the proposed technique using all the image sin Cadik dataset is 0.878, which is better than all the existing color-to-grayscale image conversion methods. Although the success rate of CP and Smith is greater than the proposed technique in case of Cadik dataset, the average C2G-SSIM index corresponding to those two techniques are lower (0.8543 and 0.8643 respectively) than that of the proposed image decolorization technique. Similarly, in the case of COLOR250 dataset, the average C2G-SSIM index is 0.912, which is better than all the existing image decolorization techniques. The implementation of the proposed method to perceptually improve the image decolorization is available in http://nlp.amrita.edu/PerceptualC2G/index.html.

4.2 Role of incorporating significant chrominance information by the proposed perceptually improved SVD image decolorization for scene classification

The experimental setup to analyze the effect of significant chrominance information on scene classification is as follows: In general, the scene classification system contains three stages namely color-to-grayscale image conversion, feature extraction and classification.

  1. Color-to-grayscale image conversion: In our experiment, the proposed color-to-grayscale image conversion to perceptually improve the existing SVD image decolorization is used in the first stage to convert the input color images to grayscale. To compare the effect of the proposed image decolorization method on scene classification, the input color images are also converted to grayscale using the most commonly used technique called the National Television Standards Committee (NTSC) rule with the ratio 0.3 : 0.6 : 0.1 of RGB respectively [23] and the existing method called SVD image decolorization [20].

  2. Feature Extraction: The dense scale invariant feature transform (SIFT) which is the suitable feature space for scene classification is used in the present work [10, 5]. Each grayscale converted input image of size is divided into grids of size with 8 pixel spacing between the grids. Hence, there are 961 grids extracted from each converted grayscale image. Each grid is represented by a SIFT feature descriptor of length 128. Thus, each grayscale converted input image is represented by feature matrix of size .

  3. Classification: The two different types of classifiers namely Adapted Gaussian Mixture Models combined with Support Vector Machines (AGMM-SVM) and AGMM combined with Deep Belief Networks and SVM (AGMM-DBN-SVM) proposed by Mandar et al. [5] & Sowmya et al. [19] respectively are used in the present experimental work.

The experimental setup for AGMM-SVM classification system is followed from [5]. The number of gaussian mixtures used to adapt the mean SIFT feature matrix of each class is 1024. The mean adapted SIFT features of each class of dimension is used as dictionary to derive the histogram feature representation of each converted grayscale image. The adapted gaussian mixture which is closest to each of the SIFT feature descriptors of an image is computed whose dimension is with the values ranging from 1 to number of mixtures (1024). The histogram is calculated for the above obtained feature vector. The normalized histogram vector is computed with respect to the dictionary of all the classes involved in the scene classification. The average histogram vector is computed and is fed as input to the SVM classifier.

In case of AGMM-DBN-SVM system, the average histogram feature vector corresponding to all the converted grayscale images derived from training set are used to learn the DBN network proposed in [19]

. The DBN network used in the present experimental work consists of two hidden layers apart from the input and the output layer. The output layer of DBN are used as input features to the SVM classifier. The two hidden layers contains 30 and 99 neurons respectively for standard color-to-grayscale image conversion (NTSC rule)

[19]. The number of neurons present in the two hidden layers for the existing SVD image decolorization are 88 and 100 respectively [19]. The architecture of DBN network for the proposed image decolorization is same as that of the existing SVD image decolorization method.

The standard Oliva Torralba (OT) scene dataset containing 8 different classses namely opencountry, coast, forest, highways, insidecity, mountain, street and tallbuildings is used for the experimental work [16, 22, 10, 5, 7]. 100 images from each class are used for training and all the remaining images are used for testing.

Image Decolorization
Methods
opencountry coast forest highways insidecity mountain street tallbuilding overall accuracy
NTSC rule 71.9 80.0 94.3 76.9 79.3 82.1 90.1 82.8 81.8
SVD 72.3 79.6 94.3 76.3 76.9 81.8 89.6 84.4 81.6
Proposed
(Modified SVD)
71.9 78.5 93.9 73.8 80.8 83.9 90.1 85.2 82.0
Table 5: Comparison of classwise accuracy (%) and overall accuracy (%) for AGMM-SVM scene classification system formed using the proposed and the existing image decolorization methods
Image Decolorization
Methods
opencountry coast forest highways insidecity mountain street tallbuilding overall accuracy
NTSC rule 90.9 92.7 92.1 87.5 93.3 95.9 96.4 97.7 93.5
SVD 91.6 93.5 94.7 87.5 91.4 95.3 96.9 96.9 93.6
Proposed
(Modified SVD)
94.2 96.9 95.2 91.3 92.8 96.4 97.4 97.3 95.3
Table 6: Comparison of classwise accuracy (%) and overall accuracy (%) for AGMM-DBN-SVM scene classification system formed using the proposed and the existing image decolorization methods

The classification assessment measure obtained for both the scene classification systems shows that the proposed image decolorization method to perceptually improve the performance of the existing SVD image decolorization technique has slightly improved the overall accuracy of the AGMM-SVM scene classification system (81.8% for NTSC , 81.6% for SVD and 82.0% for the proposed method (Modified SVD) tabulated in Table 5) and has significantly improved the overall classification accuracy of the AGMM-SVM-DBN scene classification system (93.5% for NTSC , 93.6% for SVD and 95.3% for the proposed method (Modified SVD) tabulated in Table 6.

The class accuracy obtained using the proposed image decolorization method which are greater than that of the most commonly used NTSC rule and the existing SVD image decolorization for both the scene classification systems are marked as bold in Table 5 & 6. The improvement in the overall classification accuracy of the AGMM-SVM scene classification system by the proposed image decolorization method is due to the improvement in the class accuracy namely insidecity, mountain street and tall building (Table 5) . Similarly, the significant improvement in the overall classification accuracy of the AGMM-DBN-SVM scene classification system by the proposed image decolorization method is due to the improvement in the classification accuracy of 6 classes out of available 8 classes (Table 6).

The significant improvement in the AGMM-DBN-SVM scene classification system by the proposed image decolorization is due to the training of deep belief networks using the dense SIFT feature descriptors which extracted the significant chrominance information with the improved perceptual quality in the grayscale images converted using the proposed image decolorization method.

4.3 Model level combination of the proposed image decolorization method and the most commonly used NTSC rule

The analysis of classification accuracy obtained for each class shows the presence of complimentary class information provided by the proposed image decolorization method with respect to the standard NTSC rule. i.e., there are significant number of images misclassified by the standard NTSC rule are correctly classified by the proposed method for image decolorization and vice-versa. This motivated to combine the complimentary class information provided by the proposed image decolorization with the standard NTSC rule for the AGMM-DBN-SVM scene classification system whose, overall classification accuracy for all the image decolorization methods are greater than the AGMM-SVM image classification system.

Thus, the complimentary class information are combined based on the maximum probability score computed by the AGMM-DBN-SVM models for NTSC rule and the proposed method respectively. The class label associated with the model corresponding to the maximum probability score computed using the Eq. (

8) is selected as the class information of the input image.

(8)

where, represents the dense SIFT feature vectors of an image, and represents the AGMM-DBN-SVM models obtained using NTSC rule and the proposed method respectively.

Image Decolorization
Methods
opencountry coast forest highways insidecity mountain street tallbuilding overall accuracy
NTSC rule 90.9 92.7 92.1 87.5 93.3 95.9 96.4 97.7 93.5
NTSC & SVD 91.3 93.1 94.7 88.8 93.8 95.9 95.8 98.1 94.1
Proposed
(NTSC & Modified SVD)
91.9 94.6 95.2 91.3 95.2 96.7 96.9 98.1 95.0
Table 7: Comparison of classwise accuracy (%) and overall accuracy (%) for the model level combination

The classification accuracy obtained for each class and the overall accuracy obtained for the AGMM-DBN-SVM model level combinations of the proposed image decolorization and the most commonly used NTSC rule is tabulated in Table 7. The classification accuracy of all the classes obtained using NTSC rule are improved by the above defined model level combination which led to the significant improvement in the overall classification accuracy of the NTSC rule from 93.5% to 95.0% (Table 7).

The overall classification accuracy of the AGMM-DBN-SVM system for NTSC rule is improved from 93.5% to 94.1% when the complimentary class information provided by the existing SVD image decolorization is combined with the NTSC rule. But, the improvement in the overall classification accuracy of the NTSC rule is greater (95% (Table7)) when the complimentary class information provided by the proposed image decolorization method is combined with the NTSC rule. This is because of the improvement in the classification accuracy of all the classes due to the presence of complimentary class information captured by the proposed method of image decolorization. Thus, the effectiveness of the proposed method to perceptually improve the performance of the existing SVD image decolorization is also proved through the classification measures obtained for the model level combination system.

5 Summary and Conclusion

In the present work, a perceptually improved image decolorization technique using C2G-SSIM and SVD is proposed. The proposed method is experimented on two standard color-to-grayscale image conversion datasets namely Cadik and COLOR250. The effectiveness of the proposed image decolorization method is compared against 8 benchmark techniques based on the image quality assessment metric for image decolorization called C2G-SSIM, success rate.

The experimental results analysis based on C2G-SSIM and success rate shows that the proposed color-to-grayscale image conversion algorithm performs better than all the existing benchmark techniques for image decolorization. Also, the role of incorporating significant chrominance information by the proposed image decolorization method for scene classification is confirmed by the improvement in the scene classification accuracy obtained for two different scene classification systems called AGMM-SVM and AGMM-DBN-SVM developed using dense scale invariant feature transform (SIFT) as features.

Also, a new method is devised to combine the models obtained using the proposed image decolorization method with the standard color-to-grayscale image conversion technique which improved the overall performance of the scene classification system. The overall scene classification accuracy obtained using the newly devised model level combination of the proposed image decolorization and the standard color-to-grayscale image conversion is better than the classification accuracy obtained using the standard image decolorization method independently.

The main contributions of the present work are as follows:

  • Developed a new algorithm to improve the perceptual quality of the image decolorization.

  • Significant improvement in the scene classification system using the dense SIFT features extracted from the grayscale images converted by the proposed image decolorization method as input to the AGMM-DBN-SVM classifier.

  • The combination of the AGMM-DBN-SVM models of the proposed image decolorization technique with the most commonly used NTSC rule for image conversion improved the scene classification performance of the existing standard NTSC method for image decolorization.

As the future scope of the present work, the performance of scene classification by the proposed image decolorization method can be analyzed at feature level using the different existing benchmark feature extraction techniques for scene classification.

References

  • [1] Ali Alsam and Mark S.Drew. Fast multispectral2gray. Journal of Imaging Science and Technology, 53 (6):60401–1, 2009.
  • [2] Amy.A.Gooch, Sven.C.Olsen, Jack Tumblin, and Bruce Gooch. Color2gray: salience-preserving color removal. ACM Trans. on Graphics (TOG), 24(3):634–639, 2005.
  • [3] CewuLu, LiXu, and Jiaya Jia. Contrast preserving decolorization. In Proc. IEEE Int. Conf. Computational Photography (ICCP), pages 1–7, 2012.
  • [4] CeWuLu, Li Xu, and Jiaya Jia. Contrast preserving decolorization with perception-based quality metrics. Int. J. Comput Vis, 110(2):222–239, 2014.
  • [5] Mandar Dixit, Nikhil Rasiwasia, and Nuno Vasconcelos. Adapted gaussian models for image classification. In

    proc. Int.Conf.on Computer Vision and Pattern Recognition (CVPR)

    , pages 937–943, 2011.
  • [6] Zhou Bolei et al. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 1(1):487–495, 2014.
  • [7] Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In proc. Int. Conf. on Computer Vision and Pattern Recognition, (CVPR), pages 524–531, 2005.
  • [8] Mark Grundland and Neil A. Dodgson. Decolorize: Fast, contrast enhancing, color to grayscale conversion. Int. J. Pattern Recognit., 40(11):2891–2896, 2007.
  • [9] Yongjin Kim, Cheolhun Jang, Julien Demouth, and Seungyong Lee. Robust color-to-gray via nonlinear global mapping. ACM Trans. on Graphics (SIGGRAPH ASIA ’09), 28(5):161–164, 2009.
  • [10] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 2169–2178, 2006.
  • [11] Chun Wei Liu and Tyng Luh Liu. A sparse linear model for saliency-guided decolorization. In Proc. Twentieth IEEE Int. Conf. Image Processing (ICIP), pages 1105 – 1109, 2013.
  • [12] Cewu Lu, Li Xu, and Jiaya Jia. Real-time contrast preserving decolorization. In proc. Int. Conf. Comput. Graphics and Interactive Techniques (SIGGRAPH), pages 34:1–34:4, 2012.
  • [13] Kede Ma, Tiesong Zhao, Kai Zeng, and Zhou Wang. Objective quality assessment for color-to-gray image conversion. IEEE Transactions on Image Processing, 24(12):4673–4685, 2015.
  • [14] M.Cadik. Perceptual evaluation of color-to-grayscale image conversions. Comput. Graphics Forum, 27(7):1745–1754, 2008.
  • [15] L. Neumann, M.Cadik, and A. Nemcsics. An efficient perception-based adaptive color to gray transformation. In Proc. Third Eurographics conf. Computational Aesthetics in Graphics, Visualization and Imaging, pages 73–80, 2007.
  • [16] Aude Oliva and Antonio Torralba. Modelling the shape of the scene: A holistic representation of the spatial envelope. Int.J.Computer Vision, 42(3):145–175, 2001.
  • [17] Karl Rasche, Robert Geist, and James Westall. Re-coloring images for gamuts of lower dimension. Int. J. Comput. Graphics Forum, 24(3):423–432, 2005.
  • [18] Kaleigh Smith, Pierre-Edouard Landes, Joelle Thollot, and Karol Myszkowski. Apparent greyscale:a simple and fast conversion to perceptually accurate images and video. Int. J. Comput. Graphics Forum, 27(2):193–200, 2008.
  • [19] V. Sowmya, Aleena Ajay, D. Govind, and K. P. Soman. Improved color scene classification systemusing deep belief networks and support vector machines. In Proc. IEEE Int.Conf.on Signal and Image Processing Applications (ICSIPA) (Accepted), 2017.
  • [20] V. Sowmya, D. Govind, and K. P. Soman. Significance of incorporating chrominance information for effective color-to-grayscale image conversion. Int. J. Signal, Image and Video Processing, 11(1):129–136, 2017.
  • [21] Alexander Suhre, Kivanc Kose, A.Enis Cetin, and Metin N.Gurcan. Content-adaptive color transform for image compression. In Proc. Seventeenth Int. Conf. Image Processing, 2010.
  • [22] Zhao Xie, Ran Ling, Kewei Wu, and Jun Gao. Learning robust independent bases for accurate scene categorization. In proc. Int. Congress on Image and Signal Processing (CISP), pages 459–463, 2012.
  • [23] Y.C.Faroudja. NTSC and beyond. IEEE Trans.Consum.Electron., 34:166–178, 1988.