A laboratory-created dataset with ground-truth for hyperspectral unmixing evaluation

02/22/2019 ∙ by Min Zhao, et al. ∙ IEEE 14

Spectral unmixing is an important and challenging problem in hyperspectral data processing. This topic has been extensively studied and a variety of unmixing algorithms have been proposed in the literature. However, the lack of publicly available dataset with ground-truth makes it difficult to evaluate and compare the performance of unmixing algorithms in a quantitative and objective manner. Most of the existing works rely on the use of numerical synthetic data and an intuitive inspection of the results of real data. To alleviate this dilemma, in this study, we design several experimental scenes in our laboratory, including printed checkerboards, mixed quartz sands, and reflection with a vertical board. A dataset is then created by imaging these scenes with the hyperspectral camera in our laboratory, providing 36 mixtures with more than 130, 000 pixels with 256 wavelength bands ranging from 400nm to 1000nm. The experimental settings are strictly controlled so that pure material spectral signatures and material compositions are known. To the best of our knowledge, this dataset is the first publicly available dataset created in a systematic manner with ground-truth for spectral unmixing. Some typical linear and nonlinear unmixing algorithms are also tested with this dataset and lead to meaningful results.



There are no comments yet.


page 1

page 4

page 6

page 7

page 8

page 11

page 12

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Hyperspectral imaging is a continuously growing field of study and has received considerable attention in the last decade. Hyperspectral data provide a wide spectral range, coupled with a high spectral resolution. These characteristics are suitable for the detection and classification of surfaces and chemical elements in the observed images. Rich information in the spectral dimension provides solutions to many problems that cannot be solved by traditional RGB imaging or multispectral imaging. However, because of factors such as the low spatial resolution of the spectral imaging devices, and the diversity of materials in a scene, an observed pixel may contain several materials. Spectral unmixing plays an important role in hyperspectral data processing, aimed at separating a pixel spectrum into a set of spectral signatures, termed endmembers, and a set of fractional abundances associated with each endmember  [2].

A significant amount of effort has been made in the past decade to solve the spectral unmixing problem. Endmember identification and abundance estimation can be conducted either in a sequential or a simultaneous manner. Sequential techniques first determine the endmembers, endmember extraction algorithms such as purity index algorithm (PPI) 

[3], vertex component analysis (VCA) [4] and N-FINDR algorithm [5]. In the sequel, the problem boils down to determination of the fractional abundances [6]. Simultaneous techniques determine the endmembers and the estimated abundance fractions at the same time, mainly by using matrix decomposition methods [7], sparse learning methods [8], or optimization methods [9] to this purpose. Moreover, several other facts have been further taken into account to enhance the modeling capacity and improve the performance of the unmixing algorithms. As hyperspectral images inherently contain two-dimensional information, it is natural to impose the spatial continuity on the estimated abundances [10, 11]. Considering the limitation of the linear unmixing due to the multilayer reflectance and the intimate interaction of materials, several works have investigated the nonlinear models and nonlinear unmixing algorithms [12, 13, 14, 15]. Taking into account the endmember variability in the observed scenario, researchers have considered a bundle of spectra to be signatures of a material during the unmixing process [16, 17, 18].

In contrast to the increasing development of various types of unmixing algorithms, the way to perform an effective and objective evaluation of these algorithms remains restrictive and unconvincing, although both synthetic data and real data are used in the literature. Compared with the target detection and classification tasks where several labeled datasets are available, the lack of ground-truth information for the unmixing makes it difficult to evaluate and compare the algorithms in an objective manner. On one hand, synthetic data are generated by the assumed mathematical models. Validating an algorithm with the data generated by the same model restricts the evaluation of the general performance of the algorithm. On the other hand, as there is no ground-truth dataset for the spectral unmixing task, it is difficult to interpret the unmixing results of the algorithms even though they have been applied to real datasets. This restricts a comparison of the results on the basis of the observation and the intuitive interpretation of the estimated abundance maps. See Sec. II-B for a more detailed review of and discussion on this issue.

Many hyperspectral datasets are captured by airborne devices and satellites. Labeling these data with material distribution fractions is extremely difficult. In this work, we design several experimental scenes in our laboratory with controlled settings, where the pure material spectra and the material composition methods are known. We then capture these scenes with a hyperspectral device and perform the necessary preprocessing to generate the dataset. To the best of our knowledge, this dataset is the first publicly available dataset with ground-truth for spectral unmixing created in a systematic manner. A summary of our contributions and the main properties of the data as follows:

  • We build a laboratory-created dataset with ground-truth for evaluating hyperspectral unmixing algorithms. This dataset is available on the corresponding author’s website http://www.jie-chen.com/.

  • We design several scenes to model different mixture mechanisms, including printed checkerboards, mixed quartz sands, and reflection with a vertical board. Both the images of pure materials and the composition of each material are provided so that researchers have access to the endmembers and the corresponding abundances. The endmember variability and nonlinear mixture effects can also be observed with the created data. See Sec. III for further details on the settings.

  • We apply several typical unmixing techniques to these data. We find that linear unmixing algorithms result in abundance RMSEs of the order of for checkerboard-type data, and the nonlinear unmixing algorithms such as K-Hype and Hapke are beneficial for intimately mixed materials.

Note that this laboratory-created dataset does not have the exact same properties as real airborne and satellite data. However, it is expected to help in reducing the gap between the theoretical research and practice, and to help researchers to evaluate their unmixing algorithms in a quantifiable and comparable manner.

The rest of this paper is organized as follows. Section II reviews several popular spectral mixing models, and presents the status and limitations of the current methods of evaluating the unmixing performance. Section III presents the properties, capturing settings, and designs scenes of our created dataset in detail. In order to validate our dataset, we apply several typical endmember extraction algorithms and unmixing techniques to the created scenes; this is described in Section IV. Section V concludes the paper and presents the discussion.

Notation. Normal font and denote scalars. Boldface small letters

denote vectors. All vectors are column vectors. Boldface capital letters

denote matrices. Specifically, let be an observed column pixel, supposed to be a mixture of endmember spectra, with the number of spectral bands. Assume that is the target endmember matrix, where each column is an endmember spectral signature. For the sake of convenience, we shall denote by the -th row of , that is, the vector of the endmember signatures at the -th wavelength band. Let be the abundance column vector associated with the pixel .

Ii Existing mixing models and unmixing performance evaluation

In this section, we will briefly review several widely used mixing models and summarize the usual methods of evaluating the unmixing performance in the literature. We shall see that these models and restrictions in the existing evaluation methodology considerably motivate the experimental settings in this work.

Ii-a Widely used mixture models

Ii-A1 Linear mixture model

The linear mixture model (LMM) is widely used to identify and quantify pure components in remotely sensed images due to its simple physical interpretation and trackable estimation process. If the multiple scattering among distinct endmembers is negligible and distinct materials are considered to be partitioned in a checkerboard manner, the spectrum of a low resolution pixel is approximated by a linear mixture of endmember spectra weighted by the corresponding fractional abundances[2][19][20], namely, the observed , is given by


with being a modeling noise. Specifically, since components of represent fractions of each material, it is often considered that they satisfy the non-negativity (ANC) and sum-to-one constraints (ASC):


Although the linear mixture model has obvious practical advantages, there are many situations in which it may not be appropriate and can be advantageously replaced by a nonlinear one. Some typical ones are presented here-below.

Ii-A2 Bilinear mixing model

Considering that multiple scattering effects can be observed on complex vegetated surfaces where it is assumed that incident solar radiation is scattered through multiple bounces involving several endmembers. Bilinear model accounts for presence of second-order photon interactions by introducing additional interaction terms in the linear model. The model is given by:


where denotes the element-wise product of two vectors. Relating and with different constraints leads a variety of bilinear models [12].

Ii-A3 Linear mixing/nonlinear fluctuation model

In our previous work[13], we do not restrict to use the extra polynomial terms to describe the nonlinearity as in the bilinear model. We assume that the mixing mechanism can be described by a linear mixture of endmember spectra, with additive nonlinear fluctuations that allows to take complex interactions of endmembers into account.


where can be any real-valued functions on a compact , of a reproducing kernel Hilbert space with being its reproducing kernel. Selecting kernels allows to capture the different types of nonlinearity of the mixture model. Performing the unmixing with (6) suggests to simultaneously estimate the abundance and the nonlinear fluctuation with the so-called kernel trick. Introducing the abundance in the nonlinear term with a post-nonlinear form, we further have


This model was proposed in[21]. In[22][23], the term is also called the residual term and the associated algorithm is called the residual component analysis.

Ii-A4 Intimate model

Intimate mixtures are mixtures of grains or particles that are in close contact with each other. Light typically interacts multiple times with the particles making up the mixture before reaching the observer. The optical characteristics of such mixtures depend on many parameters, such as the total number and the fractions of each component, the grain size distribution of each component, the shape and orientation of the grains, the average optical distance between reflections, the absorption and scattering characteristics, etc. Model proposed by Hapke [24] is a popular model that describes the optical characteristics of intimately mixed materials. It can be viewed as a post-nonlinear model applied to a linear mixture in the albedo domain. See [24] for more details. Though the intimate model has a clear physical interpretation, few works perform unmixing with this model due to its complexity and requirement of several parameters of the scene.

Ii-B Evaluation of the unmixing performance

In contrast to the increasing development of various types of unmixing algorithms, the way to perform an effective and objective evaluation of algorithms remains restrictive and unconvincing. Most works in the literature evaluate the unmixing performance with both synthetic data and real data, however, with significant limitations.

Ii-B1 Evaluation with synthetic data

Synthetic data are generated using an assumed mathematical model. The use of synthetic data is considered as a baseline to evaluate and compare the unmixing algorithms, as all information, including endmembers, abundances, and the mixing model, are known. Therefore we can evaluate algorithms with objective measures. However, it is clear that the use of synthetic data is very restrictive because of the following reasons:

  • Each model has its own limitation in modeling a real mixture in the observed scene. It is not trivial to predict to the performance of an algorithm (particularly nonlinear ones) in real applications, even though it works well with the synthetic data generated using a mathematical model.

  • It is common in the exiting literature to validate an algorithm with data generated by the same model. It is not convincing to evaluate the performance of an algorithm in such a manner. For instance, an algorithm carefully devised from the bilinear model may outperform other nonlinear unmixing algorithms when we use data generated by the same model, however, it does not make much sense if the performance of this algorithm is severely biased when the practical mixture is not ideally bilinear.

Ii-B2 Evaluation with real data

Besides using synthetic data, most of the previous studies also use real images such as the Cuprite dataset [13, 20, 14, 25, 26] to validate the proposed algorithm. However, people are currently faced with a dilemma of using these real datasets captured by airborne methods. On one hand, while considering the limitation of synthetic data, evaluating algorithms with real data is indeed important. On the other hand, it is very difficult to compare the obtained results because the lack of the associated ground-truth information for real data precludes a quantitative and objective evaluation of algorithms. Researchers make the following efforts to evaluate the unmixing performance while using real data:

  • Interpreting results with intuitive observation: When testing unmixing algorithms with real data, most of the existing works [21, 20, 12, 27, 26, 25, 28, 29] illustrate the results in a straightforward manner by showing the estimated abundance maps and making visually intuitive comparisons. Clearly, this qualitative method is not very helpful to understand the performance of these algorithms.

  • Comparing reconstruction errors: Besides an intuitive examination of abundance maps, the spectral reconstruction error (RE) and the spectral angle (SA) between the observed pixel and the reconstructed one are frequently used as quantitative measures in the literature [12, 23, 11, 28]. However, it is widely accepted that the quality of reconstruction is not necessarily in proportion to the quality of unmixing, particularly for real images where a complex nonlinear mixture is involved [13, 22, 14].

  • Comparing results with classification results: In contrast to the unmixing task, there exist several datasets with ground-truth labels for the hyperspectral classification. Some works benefit from this information to evaluate the unmixing result in an indirect manner. In [30, 11]

    , the authors consider the abundance obtained from unmixing algorithms as the feature of a classifier, and then compare the classification results with the ground-truth. In 

    [10, 31, 32, 33, 29], the authors compare the abundance maps with the classification results obtained using the Tricorder software.

  • Comparing results with those obtained from existing algorithms: A handful of works consider the results from a given unmixing algorithm as the standard reference, and then compare the performance of other algorithms with it. For instance, in [34], the authors use this methodology to generate “labeled” unmixing data. In [35, 36], the authors evaluate the results of NMF by comparing them with those of FCLS.

It is clear that none of the above methods provide a convincing quantitative performance evaluation of unmixing algorithms. The result illustration with real data in the existing works is therefore not highly informative and persuasive. This motivates us to create a proper dataset with ground-truth in our laboratory for a fair evaluation of unmixing algorithms.

Fig. 1: Data capture device and schema: (a) Imaging device (Gaia Sorter). (b) Details in the imaging obscura. (c) A diagram of data acquisition setting.

Iii Description of the dataset

Iii-a General description and properties of the dataset

Our dataset possesses the following properties:

  • A variety of mixing scenes: We design several different scenes to mimic various mixture models with experiment parameters being strictly controlled, including checkerboard-type data, mixed quartz sands, and reflection with a vertical board. A single scene consists of up to 4 materials and 14 different mixtures.

  • Available ground-truth: Both material endmembers and their associated abundances are known. We image pure materials for collecting the endmembers, and use the prior known compositions or high-resolution images to provide the abundance fractions. With this information, both of the endmember extraction and the abundance estimation algorithms can be evaluated.

  • Spatial property: A raw image has a spatial resolution of pixels. In order to provide data with uniform illumination and remove unrelated objects (e.g., conveyer background, calibration board, the container), we keep the region of pixels from the center of each scene. Both raw images with considerably more pixels and the clipped images are provided in our dataset, where the clipped images already provide at least 3600 pixels with ground-truth.

  • Spectral property: The data have 256 channels in the spectral dimension, ranging from to , with a spectral resolution of up to . The noise of each band is not uniform. First and last several bands are much noisy due to the low sensitivity of the spectrometer at these two ends. These bands can simply be discarded, or kept for studying the band selection, robust unmixing, etc.

Iii-B Methods

Iii-B1 Hyperspectral device description and settings

Our data were collected by the GaiaField and GaiaSorter systems in our laboratory. Our GaiaField (Sichuan Dualix Spectral ImageTechnology Co. Ltd., GaiaField-V10) is a push-boom imaging spectrometer with a HSIA-OL50 lens, covering the visible and NIR wavelengths ranging from to , with a spectral resolution of up to . This camera adopts CCD instead of CMOS detector, leading to a lower noise level. Provided that the camera is not running in an environment with a very high temperature, the heating effect, readout noise and dark current noise will be stable. The dark current has a negligible effect for the visible and NIR range. GaiaSorter sets an environment that isolates external lights, and is endowed with a conveyer to move samples for the push-boom imaging. Conveyor belt is made of aluminum with the oxidizing blackened surface. It can be considered as a black-body in the most range of . Four tungsten-bromine lamps were used to form a hemispherical-directional illumination that simulated uniform real-world solar illumination. A halogen lamp is characterized by its stability in illumination. After a warm-up time of , the spatial illumination uniformity of our lamps is greater than and the light sources are proved to be stable over time. The distance between the camera and the samples was adjustable. The exposure time, moving speed of the conveyer, and focusing were automatically computed and controlled by the associated software that was run on a computer.

In our experiments, the camera was right above the target. The distance between the lens and the samples was adjusted to , and the field angle of the camera was . Therefore, the width of view was . The vertical distance between the lamps and the samples was approximately . At these settings, a collected raw hyperspectral cube consisted of 348 and 455 pixels in width and height, and 256 spectral bands. We then found that the resolution of a collected image is /pixel. In our experiments, we controlled the cross-sectional areas of each pure material to be less than , to ensure that each pixel was a mixture of multiple materials.

Iii-B2 Data normalization

In order to convert the collected light intensity levels to reflectance values, we applied the black-white normalization to the raw data. This normalization also aimed to remove the effect of the dark current of the camera sensor and avoided the uneven light intensity of each band. In the offline phase, the black image is acquired by turning off the light source and covering the camera lens with its cap. The white image was acquired by imaging a standard white board (foamed PEFE resin, certified by China Metrology Institute) under the same condition as that of the raw image for calibration. Then, a normalized pixel was obtained by performing the following step on the raw observation  [37]:


where and denote the spectra of the black and white frames respectively, and the difference and the division operations are applied in an element-wise manner.

Iii-C Experimental scenes and data description

Iii-C1 Scene I

In the first scene, we aimed to design a scene that matched the linear mixing model. As in the linear model, the abundance can be seen as the area percentage of each material in a pixel; we used a checkerboard image to set this scene. Three typical colors of color printing, namely, magenta, yellow, and cyan, were used as the endmembers, as shown in Fig. 

2(a). We printed these colors in a checkerboard manner consisting of a large number of or squares. The reference materials and the mixed scenes are printed on the same type of paper. This setting mimicked the low resolution data capture process and was consistent with the assumption of the linear mixture model. We prepared ten mixture settings in this way with the fraction abundances reported in Tab. I, and the mixture patterns depicted pictorially in Figs. 2(b-k). The spectral curves of the endmembers and the mixtures are illustrated in Fig. 3.

Magenta Yellow Cyan
Mixture 1 0
Mixture 2 0
Mixture 3 0
Mixture 4 0
Mixture 5 0
Mixture 6 0
Mixture 7
Mixture 8
Mixture 9
Mixture 10
TABLE I: Abundance ground-truth of Scene I.

Fig. 2: Illustration of Scene I. (a): Three pure color blocks (magenta, yellow and cyan). (b) to (k): Checkerboards consisting of or squares with different layouts to form ten mixtures. Zoomed-in part of each subfigure illustrate the sizes and arrangements of color blocks.

Fig. 3: Spectral curves of Scene I. Spectra of 50 pixels are plotted for each case, and the red curve represents the mean spectrum. (a) to (c): Spectra of pure color blocks, magenta, yellow, cyan in turn (endmembers). (d) to (m): Spectra of ten mixtures, Mixture 1- Mixture 10 in turn.

Iii-C2 Scene II

In the second scene, we mimicked the intimate mixture case by using four different colors of quartz sands. The diameter of the granules was approximately

, which was controlled by filtering through 55-mesh and 60-mesh sieves. Using sands with a uniform size allowed us to focus on the volume fractions by ruling out the factors such as the cross-sectional areas of the granules. We first prepared fourteen mixtures where the sands were uniformly distributed as shown in Fig. 

4. The percentage of each color of sands was considered as the fractional abundances with the value reported in Tab. II. The spectral curves of the endmembers and the mixtures are illustrated in Fig. 5. Thereafter, we prepared four mixtures with spatial patterns, as shown in Fig. 6. Unlike in the other experiments where abundances were given on the basis of the material composition, in this experiment, the material composition of each pixel couldnot be known immediately at this setting. Therefore, we captured high-resolution RGB images, and then analyzed the percentage of each colored sand in a low-resolution hyperspectral image with the help of the associated resolution RGB image111We captured hyperspectral images along with their associated high-resolution RGB images, and aligned them by the preset marks. Consequently, a high resolution RGB region could be linked to a hyperspectral pixel using the spatial resolution ratio. Then the proportion of the materials in a hyperspectral pixel can be evaluated by analyzing the RGB region.. This setting allowed to test algorithms that took into account the spatial correlation.

Red Green Blue White
Mixture 1
Mixture 2
Mixture 3
Mixture 4
Mixture 5
Mixture 6
Mixture 7
Mixture 8
Mixture 9
Mixture 10
Mixture 11
Mixture 12
Mixture 13
Mixture 14
TABLE II: Abundance ground-truth of Scene II.

Fig. 4: Diagram of Scene II with uniform mixtures, (a) Pure colored quartz sands. (b)-(o) Fourteen intimate mixtures.

Fig. 5: Spectral curves of Scene II: (a) to (d) Pure colored quartz sands, red, green, blue, white in turn. (e)-(r) Fourteen intimate mixturesm, Mixture 1-Mixture 14 in turn.

Fig. 6: Diagram of Scene II with quartz sands mixed with four spatial patterns (high resolution RGB images).

Iii-C3 Scene III

In this scene, we aimed to create an environment in which the second-order reflection exists. The color squares prepared in Scene I were used. Further, a plastic board was set beside the printed squares to reflect the light. A diagram of this scene is shown in Fig. 7. Eight mixtures were captured in this scene, including five mixtures with reflection, and three pure materials with reflection. The mixture fractions are reported in Tab. III, and the spectral curves of the endmembers and the mixtures are illustrated in Fig. 8.

Magenta Yellow Cyan
Mixture 1
Mixture 2
Mixture 3
Mixture 4 0
Mixture 5
Mixture 6 0
Mixture 7 1
Mixture 8 0
TABLE III: Abundance ground-truth of Scene III.
Fig. 7: Photo (left) and diagram (right) of Scene III.

Fig. 8: Spectral curves of Scene III: (a) to (d) Four endmembers, magenta, yellow, cyan, board in turn.(e)-(l) Eight mixtures, Mixture 1-Mixture 8 in turn.

Iv Algorithm tests with our dataset

In this section, we apply several typical unmixing algorithms to our dataset and examine the unmixing results. Note that the purpose of this application is to check how some existing algorithms perform with these data, rather than to provide an optimal algorithm. As it is straightforward to extract the endmember from the pure materials, we particularly focus on the unmixing with known endmembers and calculate the root mean square error (RMSE), namely,


to evaluate the abundance estimation performance of the above unmixing algorithms, where is the total number of pixels, and denote the true and estimated abundance vectors of the th pixel. Endmember extraction is also tested in the linearly mixed Scene I, as to be seen later.

Iv-a Results of Scene I

Iv-A1 Abundance estimation results of Scene I

We extracted the endmembers from the pure color blocks in Fig. 2, and then applied the fully constrained least square method (FCLS), and the nonnegative constrained least square method (NCLS) to the mixtures. The FCLS method relies on the linear mixture model and minimizes the least-square error, subject to the non-negativity and the sum to one constraints. By relaxing the sum-to-one constraint, the solution leads to the NCLS method. The mean estimates and RMSE results of FCLS and NCLS are showed in Tab. IV. We observe that both the algorithms resulted in RMSE values of the order of and the two algorithms exhibited a comparable performance. The histograms of the estimates of Mixture 2 are shown in Fig. 9.

Magenta Yellow Cyan RMSE Magenta Yellow Cyan RMSE
Mixture 1 0.16% 0.34% 0.0749
Mixture 2 0.03% 0.0155 0
Mixture 3 0.0493
Mixture 4 0.0237
Mixture 5 0.0198
Mixture 6 % 0.0239 %
Mixture 7 0.0263
Mixture 8 0.0275
Mixture 9 0.0221
Mixture 10 0.0238
TABLE IV: Abundance estimation results of Scene I. Boldface number denote the lowest RMSEs.

Fig. 9: The histogram of the estimates for Mixture 2 of Scene I: (a) FCLS, (b) NCLS, -axis is the abundance of endmember1 (yellow), -axis is the abundance of endmember2 (cyan), and -axis is the number of pixels in each interval.

Iv-A2 Endmember extraction results of Scene I

We also applied several typical endmember extraction algorithms to the mixture shown in Figs. 2(b-k) and chose two widely used metrics, namely spectral angle distance (SAD) and spectral information divergence (SID), to evaluate the similarity between a real endmember and its estimate. The following methods we tested: VCA [4], N-FINDR [5], MVC-NMF [7], MVSA [9], and SPICE [38]. Among these algorithms, the VCA and N-FINDR required the existence of pixels of pure materials, and the rest of them did not impose this requirement. Note that the scenes shown in Figs. 2(b-k) did not in fact contain pure color blocks.

The SAD and SID between the real endmember spectral signature and its estimate are respectively given by: and

with the probability distribution vector with each endmember signature

, and its estimate . The SAD and SID results of different endmember extraction methods are shown in Tab. V.

Mixture 1 10.1101 11.2335 10.1722 10.1101 9.9481 0.0527 0.0676 0.0529 0.0527 0.0517
Mixture 2 15.0557 16.0560 15.1569 15.0557 15.0027 0.1215 0.1231 0.1231 0.1215 0.1209
Mixture 3 14.1272 15.2872 14.2790 14.1272 14.1095 0.1037 0.1076 0.1056 0.1037 0.1034
Mixture 4 17.5804 20.5323 17.8193 17.5804 17.5681 0.2274 0.1985 0.2324 0.2274 0.2274
Mixture 5 18.6483 19.2550 18.6473 18.6483 18.6134 0.2358 0.2512 0.2349 0.2358 0.2358
Mixture 6 19.1086 21.3061 19.4464 19.1086 19.0999 0.2598 0.2141 0.2711 0.2598 0.2596
Mixture 7 15.7100 16.3647 16.4140 16.3326 15.4834 0.1396 0.1389 0.1409 0.1405 0.1364
Mixture 8 16.4893 17.1885 17.4584 18.2166 16.6785 0.1301 0.1312 0.1326 0.1494 0.1329
Mixture 9 16.0829 17.2970 17.8173 18.6586 15.9036 0.1591 0.1467 0.1923 0.1870 0.1533
Mixture 10 16.0358 17.5287 17.9511 18.1627 16.3448 0.1668 0.1624 0.1826 0.1890 0.1685
TABLE V: Endmember extraction results of Scene I. Boldface numbers denote the lowest SADs and SIDs.

Iv-B Results of Scene II

Iv-B1 Unmixing results of the spatially uniform mixtures

The FCLS and NCLS were firstly applied to Scene II, with the mean estimates and RMSE results reported in the first and the second columns of Tab. VI. We observe that in this scene both of these algorithms led to high estimation RMSEs, possibly caused by the fact that the intimate interaction of granules resulted in nonlinear mixture effects. Considering this, we then applied the nonlinear unmixing algorithm K-Hype to the data. The K-Hype algorithm models the observed pixel by a linear mixture and a nonlinear fluctuation of spectral signatures, and it is a kernel based algorithm to perform the unmixing, see [13] for more details. We used the Gaussian kernel, with . The following methods were also tested -linear ([39], MLM [40], Hapke model [24]. The obtained results are reported in the third column of Tab. VI and Tab. VII. We observe that K-Hype and Hapke model led to reduced RMSEs for most of the mixtures. This showed the usefulness of the nonlinear unmixing.

Red Green Blue White RMSE Red Green Blue White RMSE Red Green Blue White RMSE
Mixture 1
Mixture 2
Mixture 3 0.0334
Mixture 4
Mixture 5 0.1054
Mixture 6 0.2909
Mixture 7 0.0533
Mixture 8
Mixture 9 0.0662
Mixture 10 0.0848
Mixture 11
Mixture 12 0.0586
Mixture 13 0.0867
Mixture 14
TABLE VI: Estimation results of Scene II. Boldface number denote the lowest RMSEs in Tab. VI and Tab. VII.
-linear MLM Hapke
Red Green Blue White RMSE Red Green Blue White RMSE Red Green Blue White RMSE
Mixture 1 0.0887
Mixture 2 0.1181
Mixture 3
Mixture 4 0.0798
Mixture 5
Mixture 6 0.0905
Mixture 7
Mixture 8 0.0878
Mixture 9
Mixture 10
Mixture 11 0.0851
Mixture 12
Mixture 13
Mixture 14 0.0849
TABLE VII: Estimation results of Scene II. Boldface number denote the lowest RMSEs in Tab. VI and Tab. VII.

Iv-B2 Unmixing results of the mixtures with spatial pattern

The true abundance maps of these four mixtures (clipped center square regions) are shown in the first columns in Figs. 10 to 13. The abundance maps estimated using the FCLS, NCLS, and K-Hype algorithms are shown alongside. We observe that the general spatial patterns of the estimated maps were consistent with those of the ground-truth reference; however, the RMSEs are notable. The K-Hype algorithm showed better RMSEs than the other two algorithms in mixtures of patterns A, C, and D. A further study on these mixtures is necessary.

Fig. 10: Abundance maps of the mixture with spatial pattern-A in Scene II. Rows: two materials. Columns: ground-truth, results of FCLS, NCLS, K-Hype, with and 0.2242 respectively.

Fig. 11: Abundance maps of the mixture with spatial pattern-B in Scene II. Rows: two materials. Columns: ground-truth, results of FCLS, NCLS, K-Hype, with and 0.3543 respectively.

Fig. 12: Abundance maps of the mixture with spatial pattern-C in Scene II. Rows: three materials. Columns: ground-truth, results of FCLS, NCLS, K-Hype, with and 0.2396 respectively.

Fig. 13: Abundance maps of the mixture with spatial pattern-D in Scene II. Rows: three materials. Columns: ground-truth, results of FCLS, NCLS, K-Hype, with and 0.2235 respectively.

Iv-C Results of Scene III

In this scene, the spectra of the pure printed colors and that of the board are used as the endmembers. Then, the unmixing algorithms were executed to estimate the abundances of the five mixtures. The mean estimates are shown in Tab. VIII. We observe that the board significantly contributed to the spectra because of to its strong reflection. As it was difficult to estimate the contribution from the board, we normalized the sum of the abundances of the printed color to 1 and then compared it with the ground-truth obtained from the layouts of the checkerboards. These RMSEs are shown in Tab. IX. We found that K-Hype exhibited good performance for the mixture with multiple materials.

FCLS NCLS K-Hype Bilinear
Magenta Yellow Cyan Board Magenta Yellow Cyan Board Magenta Yellow Cyan Board Magenta Yellow Cyan Board
Mixture 1
Mixture 2
Mixture 3
Mixture 4
Mixture 5
Mixture 6
Mixture 7
Mixture 8
TABLE VIII: Abundance estimation result of Scene III.
FCLS NCLS K-Hype Bilinear
Mixture 1
Mixture 2
Mixture 3
Mixture 4
Mixture 5
Mixture 6
Mixture 7
Mixture 8
TABLE IX: RMSE results (without considering the board) of Scene III. Boldface number denote the lowest RMSEs.

V Conclusion

In our work, we created a dataset for evaluating the performance of spectral unmixing algorithms. The dataset provides 3 types of mixture scenes and 36 mixtures that simulate different mixture models. Each pixel consists of 256 spectral bands with a spectral resolution of up to . Both of the endmembers of pure materials and their compositions are priorly known, allowing us to testing unmixing algorithms in a quantitative and an objective manner. Typical unmixing algorithms, including the FCLS, NCLS, K-Hype, Bilinear, -linear, MLM, and Hapke methods were applied to this dataset and led to interpretable results. Unmixing of some complex scenes however requires some further study. In the future, we will on one hand make efforts to enrich the dataset with more settings, and on the other hand, to devise and test unmixing algorithms with these labeled data.


  • [1] M. Zhao and J. Chen, “A dataset with ground-truth for hyperspectral unmixing,” in Proc. IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Valencia, Spain, Jul. 2018, pp. 5081–5084.
  • [2] J. M. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader, and J. Chanussot, “Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches,” IEEE J. Sel. Top. Appl. Earth Observat. Remote Sens., vol. 5, no. 2, pp. 354–379, 2012.
  • [3] J. W. Boardman, “Automating spectral unmixing of aviris data using convex geometry concepts,” JPL, Summaries of the 4th Annual JPL Airborne Geoscience Workshop, vol. 1, pp. 11–14, 1993.
  • [4] J. M. Nascimento and J. M. Bioucas-Dias,

    “Does independent component analysis play a role in unmixing hyperspectral data?,”

    IEEE Trans. Geosci. Remote Sens., vol. 43, no. 1, pp. 175–187, 2005.
  • [5] M. E. Winter, “N-findr: An algorithm for fast autonomous spectral end-member determination in hyperspectral data,” in Imaging Spectrometry V. International Society for Optics and Photonics, 1999, vol. 3753, pp. 266–276.
  • [6] D. Heize and C. Chang, “Fully constrained least squares linear mixture analysis for material quantification in hyperspectral imagery,” IEEE Trans. Geosci. Remote Sens., vol. 39, no. 3, pp. 529–545, 2001.
  • [7] L. Miao and H. Qi, “Endmember extraction from highly mixed data using minimum volume constrained nonnegative matrix factorization,” IEEE Trans. Geosci. Remote Sens., vol. 45, no. 3, pp. 765–777, 2007.
  • [8] M.-D. Iordache, J. M. Bioucas-Dias, and Plaza A., “Sparse unmixing of hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 6, pp. 2014–2039, 2011.
  • [9] J. Li, A. Agathos, D. Zaharie, J. M. Bioucas-Dias, Plaza A., and X. Li, “Minimum volume simplex analysis: a fast algorithm for linear hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 53, no. 9, pp. 5067–5082, 2015.
  • [10] M.-D. Iordache, J. M. Bioucas-Dias, and A. Plaza, “Total variation spatial regularization for sparse hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 11, pp. 4484–4502, 2012.
  • [11] J. Chen, C. Richard, and P. Honeine, “Nonlinear abundance estimation of hyperspectral images with -norm spatial regularization,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 5, pp. 2654–2665, May. 2014.
  • [12] A. Halimi, Y. Altmann, N. Dobigeon, and J. Tourneret, “Nonlinear unmixing of hyperspectral images using a generalized bilinear model,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 11, pp. 4153–4162, 2011.
  • [13] J. Chen, C. Richard, and P. Honeine, “Nonlinear unmixing of hyperspectral data based on a linear-mixture/nonlinear-fluctuation model,” IEEE Trans. Signal Process., vol. 61, no. 2, pp. 480–492, 2013.
  • [14] Q. Qu, N. Nasrabadi, and T. Tran, “Abundance estimation for bilinear mixture models via joint sparse and low-rank representation,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 7, pp. 4404–4423, 2014.
  • [15] M. Tang, L. Gao, A. Marinoni, P. Gamba, and B. Zhang, “Integrating spatial information in the normalized p-linear algorithm for nonlinear hyperspectral unmixing,” IEEE J. Sel. Top. Appl. Earth Observat. Remote Sens., 2017.
  • [16] C. A. Bateson, G. P. Asner, and C. A. Wessman, “Endmember bundles: A new approach to incorporating endmember variability into spectral mixture analysis,” IEEE Trans. Geosci. Remote Sens., vol. 38, no. 2, pp. 1083–1094, 2000.
  • [17] B. Somers, M. Zortea, A. Plaza, and G. P. Asner, “Automated extraction of image-based endmember bundles for improved spectral unmixing,” IEEE J. Sel. Top. Appl. Earth Observat. Remote Sens., vol. 5, no. 2, pp. 396–408, 2012.
  • [18] M. Xu, L. Zhang, and B. Du, “An image-based endmember bundle extraction algorithm using both spatial and spectral information,” IEEE J. Sel. Top. Appl. Earth Observat. Remote Sens., vol. 8, no. 6, pp. 2607–2617, 2015.
  • [19] N. Keshava and J. F. Mustard, “Spectral unmixing,” IEEE Signal Processing Mag., vol. 19, no. 1, pp. 44–57, 2002.
  • [20] O. Eches, N. Dobigeon, C. Mailhes, and J. Tourneret, “Bayesian estimation of linear mixtures using the normal compositional model. application to hyperspectral imagery,” IEEE Trans. Image Process., vol. 19, no. 6, pp. 1403–1413, 2010.
  • [21] J. Chen, C. Richard, and P. Honeine, “Estimating abundance fractions of materials in hyperspectral images by fitting a post-nonlinear mixing model,” in Proc. IEEE. WHISPERS, 2013, 2013, pp. 1–4.
  • [22] Y. Altmann, N. Dobigeon, S. McLaughlin, and J. Tourneret, “Residual component analysis of hyperspectral images: application to joint nonlinear unmixing and nonlinearity detection,” IEEE Trans. Image Process., vol. 23, no. 5, pp. 2148–2158, 2014.
  • [23] Y. Altmann, M. Pereyra, and S. McLaughlin, “Bayesian nonlinear hyperspectral unmixing with spatial residual component analysis,” IEEE Trans. CI, vol. 1, no. 3, pp. 174–185, 2015.
  • [24] B. Hapke, “Bidirectional reflectance spectroscopy: 1. theory,” J. Geophys. Res., vol. 86, no. B4, pp. 3039–3054, 1981.
  • [25] J. Yang, Y.-Q. Zhao, J. C.-W. Chan, and S. G. Kong, “Coupled sparse denoising and unmixing with low-rank constraint for hyperspectral image,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 3, pp. 1818–1833, 2016.
  • [26] J. Sigurdsson, O. Ulfarsson, and J. R. Sveinsson, “Blind hyperspectral unmixing using total variation and sparse regularization,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 11, pp. 6371–6384, 2016.
  • [27] P. Giampouras, K. E. Themelis, A. A. Rontogiannis, and K. D. Koutroumbas, “Simultaneously sparse and low-rank abundance matrix estimation for hyperspectral image unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 8, pp. 4775–4789, 2016.
  • [28] A. Halimi, J. Bioucas-Dias, N. Dobigeon, G. S. Buller, and S. McLaughlin, “Fast hyperspectral unmixing in presence of nonlinearity or mismodeling effect,” IEEE Trans. Comput. Imag., vol. 3, no. 2, 2017.
  • [29] X. Jiang, M. Gong, H. Li, and J. Li, “A two-phase multiobjective sparse unmixing approach for hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 1, pp. 508–523, 2018.
  • [30] I. Dopido, M. Zortea, A. Villa, Plaza A., and Gamba P., “Unmixing prior to supervised classification of remotely sensed hyperspectral images,” IEEE Geosci. Remote Sens. Lett., vol. 8, no. 4, pp. 760—764, 2011.
  • [31] M.-D. Iordache, J. M. Bioucas-Dias, and A. Plaza, “Collaborative sparse regression for hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 1, pp. 1–14, 2013.
  • [32] N. Akhtar and A. Mian, “RCMF: robust constrained matrix factorization for hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 6, pp. 3354–3366, 2017.
  • [33] S. Zhang, J. Li, H.-C. Li, C. Deng, and A. Plaza, “Spectral–spatial weighted sparse regression for hyperspectral image unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 6, pp. 3265–3276, 2018.
  • [34] C. Shi and L. Wang, “Linear spatial spectral mixture model,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 6, pp. 3599–3611, 2016.
  • [35] Y. Qian, S. Jia, J. Zhou, and A. Robles-Kelly, “Hyperspectral unmixing via sparsity-constrained nonnegative matrix factorization,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 11, pp. 4282–4297, 2011.
  • [36] X. Lu, H. Wu, Y. Yuan, P. Yan, and X. Li, “Manifold regularized sparse nmf for hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 51, no. 5, pp. 2815–2826, 2013.
  • [37] S. Serranti, R. Palmieri, and G. Bonifazi, “Hyperspectral imaging applied to demolition waste recycling: innovative approach for product quality control,” J. Electron. Imaging, vol. 24, no. 4, pp. 043003, 2015.
  • [38] A. Zare and P. Gader, “Sparsity promoting iterated constrained endmember detection in hyperspectral imagery,” IEEE Geosci. Remote Sens. Lett., vol. 4, no. 3, pp. 446–450, 2007.
  • [39] A. Marinoni and P. Gamba, “A novel approach for efficient -linear hyperspectral unmixing,” IEEE J. Sel. Top. Sign. Proces., vol. 9, no. 6, pp. 1156–1168, 2015.
  • [40] R. Heylen and P. Scheunders, “A multilinear mixing model for nonlinear spectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 1, pp. 240–251, 2016.