Spectral video construction from RGB video: Application to Image Guided Neurosurgery

12/14/2016 ∙ by Md. Abul Hasnat, et al. ∙ University of Eastern Finland 0

Spectral imaging has received enormous interest in the field of medical imaging modalities. It provides a powerful tool for the analysis of different organs and non-invasive tissues. Therefore, significant amount of research has been conducted to explore the possibility of using spectral imaging in biomedical applications. To observe spectral image information in real time during surgery and monitor the temporal changes in the organs and tissues is a demanding task. Available spectral imaging devices are not sufficient to accomplish this task with an acceptable spatial and spectral resolution. A solution to this problem is to estimate the spectral video from RGB video and perform visualization with the most prominent spectral bands. In this research, we propose a framework to generate neurosurgery spectral video from RGB video. A spectral estimation technique is applied on each RGB video frames. The RGB video is captured using a digital camera connected with an operational microscope dedicated to neurosurgery. A database of neurosurgery spectral images is used to collect training data and evaluate the estimation accuracy. A searching technique is used to identify the best training set. Five different spectrum estimation techniques are experimented to indentify the best method. Although this framework is established for neurosurgery spectral video generation, however, the methodology outlined here would also be applicable to other similar research.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 10

page 11

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The objective of spectral imaging is to provide an accurate representation of the color information of an object being imaged. Spectral imaging is an imaging technnique where each pixel contains a full wavelength spectrum of all points in the image scene. Spectral image allows reproduction of color image under various illuminations, analyze images of individual spectral band and study computational color vision and color constancy Shimano et al (2007). Individual spectral band image provides better discrimination of objects of interest, segmentation, recognition and classification of objects. Spectral imaging technology has been used in various fields of applications Hardeberg (2001) (technology: automatic inspection, remote sensing; industry: paper, ink, painting, printing, wood, textile, agriculture, food, chemical and medicine; service: medical). Ongoing research work on exploring the possibility of its use in several other areas reveals the importance of spectral imaging.

The task of spectral imaging can be accomplished in various ways. One approach is to use different narrow band filters with a grayscale image capturing device. A filter wheel consisting of narrowband filters (interference filter systems Mansouri et al (2005)), an acousto-optic tunable filter (AOTF) Cheng et al (1993); Morris et al (1994), or a liquid crystal tunable filter (LCTF) Morris et al (1994) are often employed in this kind of imaging systems. Therefore, in these systems an image is captured by scanning in the spectral domain. Another approach is based on spatial scanning, where the spectral information is measured line by line Hyvarinen et al (1998). The main differences between the available spectral imaging systems are in accuracy, speed and scanning methods. The spectral and the spatial resolutions of the line scanning cameras (PGP based devices Hyvarinen et al (1998)) are usually very high and one line spectral measurement can be done rapidly. However, for capturing the whole image they are quite slow in comparison to the other imaging systems. Among the filter based devices, LCTF Morris et al (1994) and AOTF Cheng et al (1993); Morris et al (1994) systems are faster than interference filter systems.

An important drawback of the available spectral imaging devices is the speed of image acquisition. Despite the invent of many recent spectral imaging devices for faster acquisition, a near real time spectral image capturing device with reasonable spatial resolution is yet to become available. In order to overcome this limitation, several alternative approaches have been proposed Yi et al (2011); Yamaguchi (2010); Du et al (2009); Park et al (2007). One approach is to produce a custom designed spectral imager with identified spectral range of interest Yi et al (2011). Another similar approach Yamaguchi (2010) is to add spectral bands with 3-RGB bands. A different approach is to recover spectral reflectance in a scene with a conventional RGB camera using multiplex illumination Park et al (2007). Beside these techniques, there are several computational approaches to obtain spectral reflectance with the use of conventional 3 band camera through spectrum estimation Shimano et al (2007); Haneishi et al (2000); Stigell et al (2007); Heikkinen et al (2007); Cheung et al (2005); López-Álvarez et al (2007); Connah et al (2004); Imai et al (2000); Imai and Berns (1999); Morovic and Finlayson (2006); Westland et al (2012); Solli et al (2005); Shi and Healey (2002). These approaches estimate spectral reflectance using various mathematical methods with the aid of prior knowledge. The aim of these estimation approaches is to reduce the cost and complexity of the image acquisition system while preserving its colorimetric and spectral accuracy. The prior knowledge includes the information about the spectra of the standard color patches or target object, corresponding response of traditional imaging device, sensor sensitivity, illumination spectra and noise.

A reasonable number of estimation techniques have been proposed in literature and usefulness of these techniques was evaluated for variety of different purposes Shimano et al (2007); Haneishi et al (2000); Stigell et al (2007); Heikkinen et al (2007); Cheung et al (2005); López-Álvarez et al (2007); Connah et al (2004); Imai et al (2000); Imai and Berns (1999); Morovic and Finlayson (2006); Westland et al (2012); Solli et al (2005); Shi and Healey (2002). Considering the behavior of the methods and availability of required data, these methods are categorized in Shimano et al (2007); Connah et al (2004). One of the most widely used techniques is the Wiener estimation Shimano et al (2007); Haneishi et al (2000); Stigell et al (2007); López-Álvarez et al (2007), which minimizes the mean square error (MSE) between the recovered and measured spectral reflectance. In the Wiener method, estimation matrix can be computed from different set of spectra information. One set of spectral information is the reflectance spectra of training samples, the illumination spectrum and the device sensitivity Shimano et al (2007); Haneishi et al (2000). The alternative set consists of only reflectance spectra of the training set and corresponding RGB values Stigell et al (2007); López-Álvarez et al (2007). A modification of the Wiener estimation is accomplished by a technique called the Pseudoinverse transformation method Shimano et al (2007), which is also called the least square method in López-Álvarez et al (2007)

. This technique uses regression analysis

Murphy (2012)

between the known spectral reflectance and corresponding device response values. Another category of the methods are the linear methods where the spectral reflectance is represented as a linear combination of the orthonormal basis vectors

Shimano et al (2007); Connah et al (2004). Similar to the modification of the Wiener estimation method, a regression analysis is employed to modify the linear method. In this modification, a relation is established between the weight column vectors for the basis vectors and corresponding sensor responses (digital counts of the trichromatic camera). This modified method is called the Imai-Berns method Shimano et al (2007); Cheung et al (2005); López-Álvarez et al (2007); Imai et al (2000); Imai and Berns (1999). One of the major foci of the Imai-Berns

method is to determine the optimal number of channels and basis vectors for sufficient estimation. According to their suggestion, the number of channels should be the same or larger than the number of basis vectors. They tested various combinations for the number of eigenvectors and channels and found that six channels is the best compromise between accuracy and the cost of adding more sets of trichromatic channels. Shi and Healey further investigated to determine the optimal number of basis vectors

Shi and Healey (2002). In their color scanner calibration experiment, they have shown that, multiple solutions exists for same device response when the number of basis vectors is more than the number of channels. In most devices three channels are used. However, in most cases three basis vectors will not be sufficient to represent accurately a large set of reflectance functions. In order to overcome this situation, Shi and Healey proposed a new method Shimano et al (2007); Cheung et al (2005); López-Álvarez et al (2007); Shi and Healey (2002) that recovers the spectral reflectances with more basis vectors than sensors while still increasing the accuracy of the recovery.

Spectral imaging has received increasing interest in the field of medical imaging modalities Fält et al (2009); Gebhart (2006); Kinnunen et al (2010); Akbari and Kosugi (2009); Keereweer et al (2011); Antikainen et al (2011); Leitner et al (2013); Crane et al (2010); Roblyer et al (2008); Toms et al (2005). With the invent of new spectral imaging based technologies which are customized for medical imaging applications, spectral imaging has been used for imaging diabetic retinopathy Fält et al (2009), imaging neurosurgical target tissues Antikainen et al (2011), determine an adequate tumor-free margin Keereweer et al (2011), evaluating the degeneration of articular cartilage Kinnunen et al (2010), distinguishing normal and neoplastic tissues in the brain Gebhart (2006), detecting cancerous tissue Leitner et al (2013), detection of the sentinel lymph node in gynecologic oncology Crane et al (2010) and detection of oral neoplasia Roblyer et al (2008). Moreover, various use of spectral imaging technology in medical field are listed in literature Akbari and Kosugi (2009), such as evaluation of tissue oxygen, diagnosis of hemorrhagic shock and detection of chronic mesenteric ischemia. Spectral imaging technology has proven its capability to produce reliable information for the surgeons with sufficient sensitivity in order to discriminating different types of tissues. Previous studies have already shown that the delineation of neoplastic tissue from normal tissue can be enhanced by optical spectroscopy Toms et al (2005) and spectral imaging devices Gebhart (2006). In addition to this, spectral analysis can be expected to provide useful clinical information for other purposes such as monitoring of blood flow or unwanted changes in normal tissues during surgeries.

Real time image capture and visualization for intra-operative image-guided surgery Keereweer et al (2011); Leitner et al (2013); Crane et al (2010) provides significant benefit to the surgeons. It is an immense interest from the surgeons to observe spectral images real time during surgery and monitor important information such as temporal changes in the organs and tissues as well as visual difference between various target tissues. Therefore, a spectral video system is necessary to incorporate during surgery. Keereweer et. al. Keereweer et al (2011) briefly discussed about the issues related to real time optical image guided surgery. They mentioned about several devices to accomplish this task where the images are considered to be taken within the range of NIR wavelengths. A multispectral real-time fluorescence imaging Crane et al (2010) is used for detecting the sentinel lymph node where the multispectral signals from different cameras are combined to yield true quantitative fluorochrome bio-distribution. Most recently, Leitner et. al. Leitner et al (2013) presents a multispectral video endoscopy system which comprises of 8 spectral band and 40 frames per second. They need to deal with image registration issues in order to obtain accurate spectral image of a moving organ or tissue. The limitations of the abovementioned real time systems are: very expensive, portability, limited number of spectral bands, additional processing of individual spectral band image and lower frame rate.

Currently no spectral video capturing device exists which has sufficiently high spatial resolution and covers a wide range of spectrum. A possible solution is that, rather than using spectral video system, a RGB video camera in cooperation with a spectrum estimation technique can be used. Then spectral video can be generated from RGB video in near real time. This research work is motivated from this solution, and proposes a framework to generate spectral video from neurosurgery RGB video. The RGB video is captured using a digital camera connected to an operational microscope dedicated to neurosurgery Antikainen et al (2011). The proposed framework successfully overcomes the limitations of the existing solutions in terms of expense, portability, range and number of spectral bands and frame rate. Moreover, we believe that the framework is extendable and customizable depending on the demand of applications.

Our key contribution in this research is to propose a complete framework for generating multispectral video from RGB video. Rather than proposing device oriented solution, our proposed solution relies on computational estimation and machine learning based approach. In order to ensure best estimate of the spectra from rgb, five different estimation methods are implemented and their performance is evaluated. The best training dataset is determined to compute the desire estimation matrix which can be used for any further estimation of the neurosurgery video frames.

We discuss the methodology of spectral estimation and experimental procedure in Section 2, then present the results and discuss about them in section 3 and finally section 4 draws conclusions.

2 Methodology

In this section, first we present several spectral estimation techniques in Sec. 2.1 and then present the complete framework and experimental procedure for spectral video generation in Sec. 2.2.

2.1 Spectral estimation methods

The basic idea of spectral estimation is to compute spectral response of a pixel value consisting RGB responses from a digital camera. The estimated spectral response will be a higher dimensional representation of the color signal. With the assumption that all surfaces are Lambertian and there is no fluorescence, the device response of a camera system with its associative reflectance under an illuminant can be modeled by Connah et al (2004):

(1)

where, is the response for the camera channel, is the spectral sensitivity of that channel, represents measurement noise and represents the wavelength. Most commonly there are 3 channels (R, G and B) of the camera, which means the usual value of =1, 2 or 3. Eq. (1) can be represented in terms of vectors and matrices as Shimano et al (2007):

(2)

where, (spectral sensitivities of sensors) is an MN matrix, (spectral power distribution of an illuminant) is a NN diagonal matrix with the samples along the diagonal, is a N1 vector with the reflectance values and is a M1 additive noise vector. Here, represents the number of samples over the visible wavelengths, and represents the number of sensors. Now, considering the camera system model in Eq. (1) and (2), our goal is to directly estimate the spectral reflectance from the device response . Next, we briefly present the details of several estimation methods that we experiment in this research.

2.1.1 Wiener estimation method

Let us consider that image pixels are measured with reflectance . After that, the same pixel values are measured with a multichannel digital camera with device response . We assume that the spectrum of the illumination , sensitivity of the multichannel digital device and associative noise of the camera are already known to us. Wiener estimation method Shimano et al (2007); Haneishi et al (2000); Stigell et al (2007); López-Álvarez et al (2007) was developed on the basis of minimum mean square error (MMSE) criterion. If we recall Eq. (2) and consider ( will be a matrix of dimension MN) then we can rewrite it as Haneishi et al (2000):

(3)

From Eq. (3), the Wiener estimation can be obtained by operating a matrix to the device response as:

(4)

were, the operator is determined such that it satisfy the MMSE criterion, i.e. it minimizes the ensemble average between the original () and estimated () spectral reflectances Haneishi et al (2000):

(5)

where, denote an ensemble average. The explicit form Shimano et al (2007); Haneishi et al (2000) of the Wiener estimation is given by:

(6)

where, denotes the autocorrelation matrix of spectral reflectances of the learning samples and denotes the autocorrelation matrix of measured noise. These autocorrelation matrices are computed as:

(7)

Eq. (6) computes the Wiener estimation matrix Shimano et al (2007); Haneishi et al (2000) considering the spectral reflectance of training spectrum (), spectrum of illumination () and device sensitivity information (). An alternative way Stigell et al (2007) to compute the estimation matrix from the spectral reflectance of training spectrum () and device response () as:

(8)

where, is a cross-correlation matrix between vectors and , and is an autocorrelation matrix of vector . These two correlation matrices are defined as

(9)

Therefore, the spectral reflectance () can be estimated by using the estimation matrix ( computed from Eq. (6) or (8)) into the Eq. (4).

2.1.2 Pseudoinverse estimation method

The idea of pseudoinverse estimation Shimano et al (2007) (also called least square López-Álvarez et al (2007)) method is to establish a direct mapping system from device response to reflectance that minimizes the least square error for a characterization or training set of known reflectance spectra with associative device responses. Let us consider, as a Mk matrix that contains the sensor responses and let be a Nk matrix that contains the corresponding spectral reflectances where is the number of learning samples. The goal of pseudoinverse method Shimano et al (2007); López-Álvarez et al (2007) is to compute a transformation matrix with dimension NM, that minimizes . The notation represents the Frobenius norm. The matrix is given by:

(10)

where represents the pseudoinverse matrix of the matrix . Once the transformation matrix is being computed, it can be further used to estimate the reflectance from device responses as: which is identical to the Eq. (4).

2.1.3 Linear estimation method

The concept of Linear estimation method Shimano et al (2007); Connah et al (2004)

came from the notion of representing the spectral reflectances as a linear combination of basis vectors. These basis vectors are derived by applying principal component analysis

Jolliffe (2002) (PCA) over the spectral reflectances. Let us consider a set of reflectance with number of samples. If we perform PCA over these reflectances then we can represent the reflectance set as:

(11)

In Eq. (11), represents the number of basis vectors which is equivalent to the number of samples in the reflectance , represent the ith basis vector and represents it associated weight, represents an Nd basis matrix containing the first basis vectors as column vectors and represents a d1 column vector that contains the

weights corresponding to the basis vectors. Let us rearrange the basis vectors in decreasing order based on their eigenvalues. If we recall Eq. (

3) without considering the noise term and substitute reflectance from Eq. (11), then we will get Shimano et al (2007):

(12)

where, is the system matrix with dimension Md computed by multiplying (matrix with dimension MN) and (matrix with dimension Nd). If we assume that number of basis vector will be equal to the number of sensors, i.e. , then is a square matrix. Therefore, the entries of matrix will be known. Now, using Eq. (12) we can compute the approximate weight column vector by rearranging it as:

(13)

Note that, if our assumption does not hold, i.e. number of basis vector is more than the number of channels, then matrix will not be a square matrix. In such situation, we have to use instead of . Here, represent the pseudoinverse Ben-Israel and Greville (2003) of matrix and represent direct inverse of matrix .

After computing the weight column vector from device responses, the estimated reflectance can be computed (by substituting Eq. (13) into Eq. (11)) as:

(14)

here, is the multichannel device response, is the corresponding estimation of spectral reflectance, is the matrix of basis vectors and is the system matrix computed from , and matrices.

2.1.4 Imai-Berns method

Imai-Berns method Shimano et al (2007); Cheung et al (2005); López-Álvarez et al (2007); Imai et al (2000); Imai and Berns (1999) follows similar concept of linear method to represent the spectral reflectances. However, it modifies linear method by employing a regression analysis between the weight column vectors and corresponding sensor responses Shimano et al (2007). Here, we assume that the basis vectors are arranged in decreasing order according to their eigenvalues Imai et al (2000); Imai and Berns (1999).

Let us consider that we have a set of learning spectral reflectances . We can represent the estimated reflectances in terms of basis vectors and weights with Eq. (11). Let us consider, be a dk matrix that contains the column vectors of the weights to represent the known spectral reflectances and be an Mk matrix that contains corresponding sensor response vectors of those reflectances. This method establishes a relationship between weights of the reflectances and corresponding device responses as using regression analysis as . Here, is a matrix with dM dimension that minimizes the Frobenius norm as:

(15)

The matrix can be used to estimate the weights from device responses as:

(16)

Finally, combining Eq. (16) and (11) we can estimate the spectral reflectance from device response as

(17)

2.1.5 Shi-Healey method

The basic idea of Shi-Healey method Shimano et al (2007); Cheung et al (2005); López-Álvarez et al (2007); Shi and Healey (2002)

is to generate unique solution while considering the number basis vectors to be more than the number of channels. Shi and Healey modified the linear method by allowing more than three degrees of freedom for accurate representation of spectral reflectances

Shi and Healey (2002).

Recall Eq. (11), and consider dividing the matrix of basis vectors into two matrices such as with N(d-M) dimension as and with NM dimension as . Applying this division into Eq. (12) and (14) the sensor response and estimated reflectance can be represented as:

(18)
(19)

where, and represent the corresponding weight column vectors for and matrices. From Eq. (18), can be solved in terms of as:

(20)

Substituting into Eq. (19), spectral reflectance can be represented as:

(21)

In general, there will be a set of spectral reflectances that are consistent with the linear method and that satisfy . Let be the spectral reflectance vector that satisfies both conditions above. Now, the weight column vector is derived from Eq. (21) as:

(22)

Therefore the estimated spectral reflectance can be obtained by the substituting Eq. (22) into equation (21) from the condition that minimizes .

An important characteristic of this method is that estimation time is proportional to the number of training reflectance spectra. For a large number of training spectra, this method is very slow because for every single pixel it has to check all the training spectra to find the best one as estimated spectra. In addition to this, for every single sensor response value in comparison with a particular training spectrum, it is necessary to investigate the correct number of basis vector which produce best estimation.

Different method requires different set of prior information. The common information required by all the methods are a set of spectral reflectance which is considered as training set. The prior knowledge required to estimate spectral reflectance by different methods discussed above is summarized in Table 1 Shimano et al (2007).

Method Name Sensitivities Illumination Reflectance RGB Values
Wiener Yes / No Yes / No Yes No / Yes
Pseudoinverse No No Yes Yes
Linear Yes Yes Yes No
Imai-Berns No No Yes Yes
Shi-Healey Yes Yes Yes No
Table 1: Requirements of prior knowledge for each method Shimano et al (2007).

Among the spectrum estimation methods discussed here, Wiener Stigell et al (2007), Pseudoinverse Shimano et al (2007) and Imai-Berns Imai et al (2000)

methods use linear regression analysis. In literature

Shimano et al (2007); Stigell et al (2007); Heikkinen et al (2007), it is suggested that, use of nonlinear regression Hardeberg (2001) with higher order (multivariate) polynomial may improve the performance of estimation. Therefore, this research considers both linear and nonlinear regression analysis for spectral estimation experiments using Wiener Stigell et al (2007), Pseudoinverse Shimano et al (2007) and Imai-Berns Imai et al (2000) method.

2.2 Spectral video generation

The aim of this research is to establish a framework which generates spectral video from neurosurgery RGB video. The complete system is called spectral video generation which is decomposed into two subtasks: (a) training spectra selection and (b) spectral video construction. Training spectra are considered as prerequisite knowledge for spectral estimation. Block diagram of spectral video generator is illustrated in Figure 1. Initially, training spectra are collected randomly from spectral images. Then representative training set is determined (Figure 1(a)) by a searching method. Figure 1(b) illustrates the process to construct spectral video with the aid of prior information. These two subtasks are briefly discussed below.

Figure 1: Block diagram of spectral video generator. (a) Training data collection (b) Spectral video generator.

Training spectra are collected from a database of spectral neurosurgery images. The database consists of 34 spectral images which were captured during surgery of 6 patients. Each spectral image was captured within the wavelength range of 420 nm to 720 nm at 10 nm interval. A spectral imaging device consisting of a camera (monochrome camera with Sony ICX414 sensor), tunable filter (VariSpecVIS VIS-10 Liquid Crystal Tunable Filter) and a focusing lens (70 mm focal length lens) was used to capture the spectral images. Since the brain tissues are moving with heart beat and breathing, each wavelength layer is slightly shifted during spectral image capture. Therefore, images must be registered accurately to achieve correct spectral responses. Among the 34 spectral images, 17 images were finally selected according to registration accuracy. Moreover, after registration few of the images contain noises and unexpected color particularly in the border. These images are manually corrected by cropping out the noisy regions. The spectral images are illustrated in Appendix A. RGB images corresponding to the spectral images were not captured through the digital video camera. However, for the purpose of estimation it is obvious to have a RGB color version of each spectral image. Therefore, the color images were generated from the spectral images considering standard illuminant D65 and CIE color matching functions. We consider these color images as training RGB images.

In a practical condition of spectral estimation, the training spectra are collected from pre-specified target points of the objects in image or from standard color patches (from calibration chart). In this research, the target points are not specified. Therefore, the training spectra are collected by choosing certain percentage (1%, 5%, 10%, 20% and 50%) of random pixels from each image. As a consequence, 5 sets of training spectra are available from each spectral image. Corresponding RGB values of the selected training spectra are also collected from RGB versions of the spectral images. The assumption of this random selection is that, spectra from all possible objects in the images will be selected. In this random selection process, total 85 (175) different set of training spectra are collected for training. However, the final goal is to select or construct only one set of training spectra which will be considered as the representative set. The representative training set is identified by observing its performance in spectrum estimation. In this research, we consider the root mean square error (RMSE) as a measure of evaluation for the training sets. RMSE is computed between original spectral image and its estimated version. The representative set of spectra may not same for all estimation methods. Therefore, the searching procedure must be applied to each estimation method separately. Steps required to search representative set of spectra are as follows:

  • Step 1: For each RGB image, estimate spectral image (Sec. 2.1) using each set of training spectra. Compute RMSE (Eq. (23)) and select the set with lowest RMSE.

  • Step 2: Using each one of the training sets (obtained from step 1), estimate all spectral images. Compute average RMSE value. Then, sort them in increasing order.

  • Step 3: Choose the first five sets and generate all possible combination. With these combinations, follow the procedure in step 2. Then, select the best training set according to lowest average RMSE value.

It is observed from the training spectra that, few of the spectra has quite unusual shapes and contains several peaks. These spectra are signifying the presence of highlight in the spectral image. In a neurosurgery spectral image, the highlights are present due to the liquid materials (blood, water) that reflects (specular reflection) the illuminant at certain extent. It is also interesting to note that most of the spectra collected from different objects have high intensity at the red wavelength range. The reason for this is the absence of blue and green colored objects in the collected spectral images. Another important property of the spectra is that they are not smooth.

The spectral video is generated from RGB video. At the beginning of the process the frames are sequentially extracted as RGB image from the video. A video frame is a RGB image which is first corrected if necessary. For example, for several estimation methods the RGB values need to be normalized within the range between 0 and 1. This correction is a part of preprocessing. After the preprocessing is completed, the spectral image is estimated (Sec. 2.1) from the RGB image with the aid of necessary prior information and particular estimation method. Then, the estimated spectral image is added to target spectral video. This process is continued until the last video frame is reached. Temporal aspect of video processing is maintained either by including wait functionality or by skipping few frames based on inter-frame similarity. The results of spectral estimation techniques are presented in the results and discussion section.

3 Results and discussion

This section evaluates the experimental performance of the spectrum estimation methods. These techniques are applied to the neurosurgery spectral image database (Appendix 1). Each spectral image has 31 spectral bands (spectral range 420 nm to 720 nm with 10 nm sampling interval).

3.1 Spectrum estimation

Five spectral estimation methods (described in Sec. 2.1) are experimented to estimate 17 neurosurgery spectral images from RGB images. Accuracy of estimation is evaluated by computing two spectral metric such as root mean square error (RMSE), goodness of fit coefficient (GFC) López-Álvarez et al (2007) and a colorimetric metric CIELAB (). The metrics are computed using the actual and estimated spectral images. With RMSE (Eq. 23) and (Eq. 25), perfect match is obtained when the value of these metrics are 0. Unlike these, in GFC (Eq. (24)) where the range of value is between 0 and 1, perfect match is obtained when the value is 1. In this research, we consider the RMSE value as the principal metric for evaluation. Additionally, we observe the values of GFC and for further evaluation if necessary.

(23)
(24)
(25)

In Eq. (23) and (24), represents the actual spectra and represents the estimated spectra. In Eq. 25, the value of , and are calculated by first converting the spectra to XYZ values and then converting XYZ to Lab. The XYZ value of standard illuminant D65 is used as a white reference in XYZ to Lab conversion.

In the evaluation process, for each method RMSE is computed for estimating 17 images using 5 set of training spectra. Therefore there are total 85 (517) RMSE values obtained. Then the average of these 85 RMSE values is calculated. This average RMSE is considered for evaluating the estimation methods (Sec. 2.1). The method that produces the lowest average RMSE is considered as the best one. Table 2 illustrates the performance of the methods in terms of average RMSE, GFC and . We consider 3 basis vectors in linear and Imai-Berns methods.

Wiener Pseudo inverse Linear Imai-Berns Shi-Healey
Average RMSE 0.029 0.029 0.177 0.036 0.082
Average GFC 0.989 0.989 0.911 0.972 0.952
Average 7.177 7.177 22.130 12.49 15.623
Table 2: Accuracy (average RMSE, average GFC and average ) of the methods.

From the results listed in Table 2, it is clear that the Wiener Stigell et al (2007) and the Pseudoinverse Shimano et al (2007) methods are producing similar estimation accuracy and they are significantly different than the Linear Connah et al (2004) and the Shi-Healey Shi and Healey (2002) methods. Accuracy produces by the Imai-Berns Imai and Berns (1999) method is nearly close to the Wiener and the Pseudoinverse methods. From different metrics of accuracy measurement it is clear that when RMSE decreases then color difference () decreases and GFC increases. This reveals that if RMSE is low, then the color difference is low and therefore the estimation accuracy is high. Because of a good agreement between the metrics, we consider only RMSE for further evaluation.

Figure 2: Plot of a recovered spectral reflectance. Comparison between actual spectra and estimated spectra from all methods.

In order to compare the estimation performance visually, we consider a pixel from a vein of the brain surface for illustration. Table 3 illustrates the performance of the methods (RMSE) for estimating the spectra of the selected pixel in terms of accuracy and speed. From the RMSE value of Table 3, it is clear that the Wiener Stigell et al (2007), Pseudoinverse Shimano et al (2007) and Imai-Berns Imai and Berns (1999) methods outperforms the other two methods. Figure 2 illustrates the recovered spectra from different methods in comparison to actual spectra. The plots clearly support the results obtained in the Table 3. In addition to the performance of estimation, the estimation time is also taken into account. Required amount of time for a particular estimation task is listed in the second row of Table 3. It is observed that estimation using the Pseudoinverse method is faster than any other methods. Therefore, in this research we consider the Pseudoinverse method for further experiments.

Wiener Pseudo inverse Linear Imai-Berns Shi-Healey
RMSE 0.0094 0.0094 0.06 0.01 0.036
Time (sec) 0.031 0.027 0.069 0.093 8.25
Table 3: Accuracy and required time of the methods for estimating a spectrum of a particular pixel.
Combination of polynomials and terms Avg. RMSE
0.019
0.018
0.019
0.02
0.018
0.018
0.017
0.017
Table 4: Accuracy (in terms of average RMSE) of Pseudoinverse method with higher order polynomials and different number of terms.

In order to further increase the accuracy of estimation using the Pseudoinverse Shimano et al (2007) method, nonlinear regression analysis Murphy (2012) have been experimented. Accuracy (in terms of average RMSE) of the combinations among different higher order polynomials in cooperation with different number of terms is presented in Table 4. For each combination, average RMSE value from 85 training sets is computed as a measure of accuracy. It is observed that the accuracy increases (average RMSE decreased) with certain combinations when the number of terms is increased. However, it is also important to note that computation time is proportional to the addition of terms. Therefore, there is a tradeoff between number of terms and computation time which should be considered depending on the demand of application. In this research, we choose the combination for neurosurgery video estimation since it has reasonable balance between computation time and accuracy.

For evaluating the accuracy of different estimation methods, the strategy is to collect five sets of training spectra from a spectral image, and then use these training sets to experiment different methods. Finally the overall accuracy of a method is computed from the average of all these (5 sets per image 17 spectral image = 85 sets of training spectra) accuracies. That means, each spectral image is estimated using the training samples collected from itself. However, in a practical application it is not feasible to determine a training dataset first and then estimate a particular spectral image. Therefore, a common dataset is necessary to estimate spectral image from any RGB image. We applied an algorithm (presented in section 2.2) to identify the best training dataset (representative set of spectra) and observed the performance. Table 5 presents the accuracy (in terms of average RMSE) of five best training datasets which are used to estimate all test spectral images. From the obtained accuracy it is observed that, there is no significant difference between these five datasets. For the purpose of selecting the representative set of spectra, we select 5% random spectra collected from image number 2.

Dataset Avg. RMSE
5%, [2] 0.026
20%, [2] 0.027
5%, [2, 12, 17] 0.03
5%, [2, 7, 12, 17] 0.028
5%, [All] 0.027
Table 5: Average RMSE of common training datasets to estimate all test images. Numbers indicate the percentage of data which has been collected from training images and numbers within ‘[]’ indicate the images which were combined.

In order to observe the estimation result in the image, a RGB image (single video frame) is extracted from a video. Then the corresponding spectral image is computed by the Pseudoinverse Shimano et al (2007) method using the training set. After that, RGB image (transformed RGB image) is computed from the estimated spectral image. The color reproduction error between the original and transformed RGB image is computed using S-CIELAB Zhang and Wandell (1997) measure. The performance of the Pseudoinverse method is evaluated using 3 terms/variables and 6 terms/variables. Corresponding S-CIELAB values are presented in table 6. From the S-CIELAB values it is observed that, performance of 6 terms estimation outperforms the performance of 3 terms estimation.

Number of terms Wiener Pseudoinverse Linear Imai-Berns
3 7.73 7.73 23.45 7.76
6 0.95 0.95 X 1.29
Table 6: Comparison (using S-CIELAB values) between original and transformed RGB images. Entry ‘X’ means not experimented.

We further investigated the error rate produced by the Pseudoinverse method. In order to identify the spectrum which produces large error, a threshold value is set as:

(26)

Based on this threshold value we identified the pixels in the image that causes high estimation error rate. The right image (b) in figure 3 indicates the regions of the pixels where the estimated error rate is above the threshold value. It is observed that, the estimation method is giving high error rate in estimating the highlighted spectra presents in the image. In the neurosurgery spectral image, these highlights appear due to the presence of liquid materials (blood, water) that reflects the illuminant at certain extent. An example of an estimated highlight spectra (with RMSE 0.43) using the Pseudoinverse Shimano et al (2007) method is illustrated in figure 3(c). From an experiment on the neurosurgery spectral image database with the threshold value (Eq. 26), it is observed that on an average 3% pixels are containing highlight in an image and causes increase of 0.003 average RMSE. However, this analysis does not reveal the actual effect of estimation where the highlight is present significantly.

Figure 3: (a) Original image. (b) black regions indicate the pixels in image that causes high estimation error. (c) Plot of an estimated spectrum of a highlighted pixel (RMSE value is 0.43).

It is observed that, if the average RMSE is computed without the highlight then it decreases from 0.26 to 0.19. For example, a particular spectral image that contains 4% highlighted pixel, increases average RMSE value 0.024. Therefore the error rate (RMSE) is proportional to the amount of highlighted pixels. We further investigated particularly on estimating the highlighted spectra. In an analysis of RMSE values for different type of spectra, it is found that when the average RMSE value in a spectral image is 0.026, the average RMSE for non highlighted pixels are 0.019 and highlighted pixels are 0.11. Therefore, it is observed that the average RMSE value for estimating only the highlight is significantly large compare to non highlighted spectra. Therefore, more analysis and experiments with different methods are necessary to reduce the estimation error rate for highlight spectra estimation. We do not consider the task of highlight estimation in the scope of this research.

4 Conclusions

This research presents a framework for generating and displaying near real time neurosurgery spectral video. Each frame in the spectral video is generated using a spectrum estimation method. Five different estimation methods are examined for estimation. The estimation matrix in Wiener method is computed using the reflectance and corresponding RGB values. In the Linear and Shi-Healey methods, standard illuminant D65 and CIE color matching function are used as illumination and device sensitivity information. Pseudoinverse and Wiener estimation methods are most accurate. It is found that second order polynomial with six terms is the best choice in terms of accuracy and computational expense. Accuracy of Imai-Berns method is nearly close to Pseudoinverse and Wiener methods. Linear estimation method does not provide an acceptable accuracy. Shi-Healey method is very slow since number of training spectra considered in this research is considerably large. Therefore, in practical application of video estimation, Shi-Healey method is not a good choice with large training set. In order to collect training spectra, the most common practice is to take the spectral image of a calibration color chart along with the target objects. However in a real surgical environment, it is not possible to place a calibration color chart and capture spectral image of the chart during surgery. Therefore, in this research the training spectra are not collected from standard target patches or recommended color charts. An alternative approach is proposed to collect training spectra. In this approach, a searching technique is applied to find out the best training set from a collection of randomly selected set of spectrum. Temporal aspect in video processing is considered. Compromise between frame rate and speed of estimation is suggested when the processor speed is not sufficient to generate near real time video.

The experimental programs in this research are written in MATLAB and the demo application is developed in C++ with OpenCV library. The methodology outlined here would also be just as applicable to other research area.

Appendix A Experimental Data

Figure 4(a) illustrates several converted (considering standard illuminant D65 and CIE color matching function) color images from the selected spectral images. Four RGB videos (captured during surgery of different patients) are available for this research. These videos were captured using a frame grabber. Each video was captured at 720576 spatial resolution and 25 frames per second. Figure 4(b) illustrates an example of a frame taken from each RGB video.

Figure 4: (a) Several representative spectral images from the database of spectral neurosurgery image. (b) Example of a frame from the four RGB videos.

References

  • Akbari and Kosugi (2009) Akbari H, Kosugi Y (2009) Hyperspectral imaging: A new modality in surgery. INTECH Open Access Publisher
  • Antikainen et al (2011) Antikainen J, von Und Zu Fraunberg M, Orava J, Jaaskelainen JE, Hauta-Kasari M (2011) Spectral imaging of neurosurgical target tissues through operation microscope. Optical review 18(6):458–461
  • Ben-Israel and Greville (2003) Ben-Israel A, Greville TN (2003) Generalized inverses: theory and applications, vol 15. Springer Science & Business Media
  • Cheng et al (1993) Cheng LJ, Chao TH, Dowdy MW, LaBaw CC, Mahoney JC, Reyes GF, Bergman K (1993) Multispectral imaging systems using acousto-optic tunable filter. In: OE/LASE’93: Optics, Electro-Optics, & Laser Applications in Science& Engineering, International Society for Optics and Photonics, pp 224–231
  • Cheung et al (2005) Cheung V, Li C, Hardeberg J, Connah D, Westland S (2005) Characterization of trichromatic color cameras by using a new multispectral imaging technique. JOSA A 22(7):1231–1240
  • Connah et al (2004) Connah D, Hardeberg JY, Westland S (2004) Comparison of linear spectral reconstruction methods for multispectral imaging. In: Image Processing, 2004. ICIP’04. 2004 International Conference on, IEEE, vol 3, pp 1497–1500
  • Crane et al (2010) Crane LM, Themelis G, Buddingh KT, Harlaar NJ, Pleijhuis RG, Sarantopoulos A, van der Zee AG, Ntziachristos V, van Dam GM (2010) Multispectral real-time fluorescence imaging for intraoperative detection of the sentinel lymph node in gynecologic oncology. Journal of visualized experiments: JoVE (44)
  • Du et al (2009)

    Du H, Tong X, Cao X, Lin S (2009) A prism-based system for multispectral video acquisition. In: Computer Vision, 2009 IEEE 12th International Conference on, IEEE, pp 175–182

  • Fält et al (2009) Fält P, Hiltunen J, Hauta-Kasari M, Sorri I, Kalesnykiene V, Uusitalo H (2009) Extending diabetic retinopathy imaging from color to spectra. In: Image Analysis, Springer, pp 149–158
  • Gebhart (2006) Gebhart SC (2006) Liquid-crystal tunable filter spectral imaging for discrimination between normal and neoplastic tissues in the brain. PhD thesis, Vanderbilt University
  • Haneishi et al (2000) Haneishi H, Hasegawa T, Hosoi A, Yokoyama Y, Tsumura N, Miyake Y (2000) System design for accurately estimating the spectral reflectance of art paintings. Applied Optics 39(35):6621–6632
  • Hardeberg (2001) Hardeberg J (2001) Acquisition and reproduction of color images: colorimetric and multispectral approaches. Universal-Publishers
  • Heikkinen et al (2007) Heikkinen V, Jetsu T, Parkkinen J, Hauta-Kasari M, Jaaskelainen T, Lee SD (2007) Regularized learning framework in the estimation of reflectance spectra from camera responses. JOSA A 24(9):2673–2683
  • Hyvarinen et al (1998) Hyvarinen TS, Herrala E, Dall’Ava A (1998) Direct sight imaging spectrograph: a unique add-on component brings spectral imaging to industrial applications. In: Photonics West’98 Electronic Imaging, International Society for Optics and Photonics, pp 165–175
  • Imai and Berns (1999) Imai FH, Berns RS (1999) Spectral estimation using trichromatic digital cameras. In: Proceedings of the International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives, vol 42
  • Imai et al (2000) Imai FH, Berns RS, Tzeng DY (2000) A comparative analysis of spectral reflectance estimated in various spaces using a trichromatic camera system. Journal of imaging science and technology 44(4):280–287
  • Jolliffe (2002) Jolliffe I (2002) Principal component analysis. Wiley Online Library
  • Keereweer et al (2011) Keereweer S, Kerrebijn JD, Van Driel PB, Xie B, Kaijzel EL, Snoeks TJ, Que I, Hutteman M, Van der Vorst JR, Mieog JSD, et al (2011) Optical image-guided surgery—where do we stand? Molecular Imaging and Biology 13(2):199–207
  • Kinnunen et al (2010) Kinnunen J, Jurvelin JS, Mäkitalo J, Hauta-Kasari M, Vahimaa P, Saarakkala S (2010) Optical spectral imaging of degeneration of articular cartilage. Journal of biomedical optics 15(4):046,024–046,024
  • Leitner et al (2013)

    Leitner R, De Biasio M, Arnold T, Dinh CV, Loog M, Duin RP (2013) Multi-spectral video endoscopy system for the detection of cancerous tissue. Pattern Recognition Letters 34(1):85–93

  • López-Álvarez et al (2007) López-Álvarez MA, Hernández-Andrés J, Valero EM, Romero J (2007) Selecting algorithms, sensors, and linear bases for optimum spectral recovery of skylight. JOSA A 24(4):942–956
  • Mansouri et al (2005) Mansouri A, Marzani FS, Hardeberg JY, Gouton P (2005) Optical calibration of a multispectral imaging system based on interference filters. Optical Engineering 44(2):027,004–027,004
  • Morovic and Finlayson (2006) Morovic P, Finlayson GD (2006) Metamer-set-based approach to estimating surface reflectance from camera rgb. JOSA A 23(8):1814–1822
  • Morris et al (1994) Morris HR, Hoyt CC, Treado PJ (1994) Imaging spectrometers for fluorescence and raman microscopy: acousto-optic and liquid crystal tunable filters. Applied spectroscopy 48(7):857–866
  • Murphy (2012) Murphy KP (2012) Machine learning: a probabilistic perspective. MIT press
  • Park et al (2007) Park JI, Lee MH, Grossberg M, Nayar SK (2007) Multispectral imaging using multiplexed illumination. In: Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, IEEE, pp 1–8
  • Roblyer et al (2008) Roblyer D, Richards-Kortum R, Sokolov K, El-Naggar AK, Williams MD, Kurachi C, Gillenwater AM (2008) Multispectral optical imaging device for in vivo detection of oral neoplasia. Journal of biomedical optics 13(2):024,019–024,019
  • Shi and Healey (2002) Shi M, Healey G (2002) Using reflectance models for color scanner calibration. JOSA A 19(4):645–656
  • Shimano et al (2007) Shimano N, Terai K, Hironaga M (2007) Recovery of spectral reflectances of objects being imaged by multispectral cameras. JOSA A 24(10):3211–3219
  • Solli et al (2005) Solli M, Andersson M, Lenz R, Kruse B (2005) Color measurements with a consumer digital camera using spectral estimation techniques. In: Image Analysis, Springer, pp 105–114
  • Stigell et al (2007) Stigell P, Miyata K, Hauta-Kasari M (2007) Wiener estimation method in estimating of spectral reflectance from rgb images. Pattern Recognition and Image Analysis 17(2):233–242
  • Toms et al (2005) Toms SA, Lin WC, Weil RJ, Johnson MD, Jansen ED, Mahadevan-Jansen A (2005) Intraoperative optical spectroscopy identifies infiltrating glioma margins with high sensitivity. Neurosurgery 57(4):382–391
  • Westland et al (2012) Westland S, Ripamonti C, Cheung V (2012) Computational colour science using MATLAB. John Wiley & Sons
  • Yamaguchi (2010) Yamaguchi M (2010) Multispectral surgery video processing and display. In: 2010 IEEE Photinic Society’s 23rd Annual Meeting
  • Yi et al (2011) Yi D, Kong L, Wang F, Liu F, Sprigle S, Adibi A (2011) Instrument an off-shelf ccd imaging sensor into a handheld multispectral video camera. Photonics Technology Letters, IEEE 23(10):606–608
  • Zhang and Wandell (1997) Zhang X, Wandell BA (1997) A spatial extension of cielab for digital color-image reproduction. Journal of the Society for Information Display 5(1):61–63