Many works have been done on salient object detection using supervised or
unsupervised approaches on colour images. Recently, a few studies demonstrated
that efficient salient object detection can also be implemented by using
spectral features in visible spectrum of hyperspectral images from natural
scenes. However, these models on hyperspectral salient object detection were
tested with a very few number of data selected from various online public
dataset, which are not specifically created for object detection purposes.
Therefore, here, we aim to contribute to the field by releasing a hyperspectral
salient object detection dataset with a collection of 60 hyperspectral images
with their respective ground-truth binary images and representative rendered
colour images (sRGB). We took several aspects in consideration during the data
collection such as variation in object size, number of objects,
foreground-background contrast, object position on the image, and etc. Then, we
prepared ground truth binary images for each hyperspectral data, where salient
objects are labelled on the images. Finally, we did performance evaluation
using Area Under Curve (AUC) metric on some existing hyperspectral saliency
detection models in literature.
Visible spectrum may contain more information than that of the colour images captured by most end-user cameras with three spectral measurements (Red-Green-Blue) for scene analysis or computer vision applications
[1]. Hyperspectral images obtained from spectral cameras provides higher spectral resolution and have spectral information on several narrow spectral bands at each pixel [1, 2, 3, 4]. Many applications in various fields (e.g. remote sensing, computer vision) have taken advantage of the spatial and spectral information of hyperspectral cameras [1]; for example, in applications such as remote sensing [5, 6, 7], scene/object analysis or object detection [3, 4, 5, 6, 7, 8, 9]
One of the possible applications of hyperspectral imagery can be salient object detection in natural scenes based on the visual attention mechanism, in which algorithms aim to explore objects or regions more attentive than the surrounding areas on the scene or images [3, 13, 14]. The first computational model of saliency detection was proposed by Itti et al. [13], which takes advantage of center-surround differences on intensity, colour and orientation features in multi-scale. Following the work [13], many works have been done on salient object detection on colour or gray images for various supervised or unsupervised applications as in [14, 15, 16, 17].
Recently, a few studies [3, 4, 7, 8] demonstrated that efficient salient object detection can also be implemented by using spectral features in visible spectrum of hyperspectral images from natural scenes. Most of these models, combine Itti et al. [13] based center-surround differences with spectral features such as spectral angle similarity or similar saliency extraction features [3, 4, 7, 8]. Regarding the evaluation data, in [7], the application is remote sensing (aerial/satellite) data, which is not the target data of this work and salient objects are labelled with bounding boxes from images. In [3], only 13 hyperspectral images (31 spectral channels with 10nm intervals in 400nm - 700 nm visible range) were used, and ground truth for salient objects were done by labelling with bounding-boxes. Yan et al. [4] use similar sources with [3] to collect data and evaluate their spectral gradient based saliency model, which are from publicly available online sources, and they improve the evaluation dataset by increasing the number to 17 images and labelling salient object with object boundaries rather than bounding box.
In summary, these models [3, 4, 8] on hyperspectral salient object detection were tested with a very few number of data selected from various online public dataset, which are not specifically created for object detection purposes. Therefore, this work aims to create a collection of larger hyperspectral image dataset from outdoor scenes that can be used for salient object detection task on hyperspectral data cubes. We aim to contribute to the field by releasing a salient object detection dataset with a collection of 60 hyperspectral images with their respective ground-truth binary images and representative rendered colour images(sRGB). We took several aspects in consideration during the data collection such as variation in object size, number of objects, foreground-background contrast, object position on the image, and etc. Then, we prepared ground truth binary images for each hyperspectral data, where salient objects are labelled with object boundaries on the images. Finally, we did performance evaluation using Area Under Curve (AUC) metric on existing hyperspectral saliency detection models from literature.
In the following section, we will explain the details of data collection process and data specifications. Then, we will demonstrate some results on the performance of salient object detection algorithms that are applicable on our dataset.
Ii Hyperspectral image dataset:
for salient object detection benchmarking
In this section, we will explain the hyperspectral image dataset for salient object detection. The dataset will be available on ”https://github.com/gistairc/HS-SOD”. For data collection, NH-AIK model hyperspectral camera is used, which is based on NH-series (NH-5) [18] (see Fig.1) and produced by Eba-Japan Co. Ltd [18]. In Table I, specifications of the camera are given.
Fig. 1: NH-AIK Hyperspectral camera used for data collection
Image resolution
1024 x 768 pixels
Measuring Wavelength
350 - 1100 nm
Wavelength resolution
5 nm
Number of spectral bands
151 channels
File format
10 bit, uint16, Band Interleaved by Line
TABLE I: NH-AIK Hyperspectral Camera Properties
The data is collected at the public parks of Tokyo Waterfront City in Odaiba, Tokyo, Japan (see the green areas in Fig.2 [19]) with the permission of Tokyo Port Terminal Corporation [20]. We collected data in several days between August - September 2017 when the weather is sunny or partially cloudy. At each data collection day, a tripod was used to fix camera to minimize motion distortion on the images. We tried to keep the exposure time and gain for camera settings fixed as much as possible depending on the daylight conditions while keeping saturation of pixels values or image visibility in mind. As a reference to the dataset users, we are providing camera settings such as exposure time and gain values for each image in a text file with the corresponding data. We also did not apply normalization on captured bands. It may improve the quality of the hyperspectral images with higher colour contrast between foreground and background regions; however, it may also decrease the difficulty of dataset for benchmarking on salient object detection task.
Fig. 2: Parks (green areas) at Tokyo Waterfront City (Odaiba, Tokyo) visited for data collection
After obtaining various hyperspectral images, we have selected 60 images from approximately fifty different scenes with the conditions: i) we removed distorted images due to motion in the scene (depending on the exposure time, one image may take a few seconds for camera), ii) we considered several aspects such as variations in salient object size, spatial positions of objects on images, number of salient objects, foreground-background contrast , iii) a few images has the same scene but the object positions, object distance, or number of objects varied.
Fig. 3: Sample images of scenes from hyperspectral dataset rendered in sRGB and respective ground truth binary images for salient objects
For the convenience of salient object detection task, we cropped spectral bands around the visible spectrum and we saved hyper-cubes for each scene in ”.mat” file format after sensor dark-noise correction. As defined in [21], visible spectrum has a well accepted range of 380 - 780 nm though the range between 400 - 700nm as in [3, 4] may also be used. To keep the range wide and flexibility to the people who want to use the dataset, we selected the defined range of 380 - 780 nm in [21] for our dataset though visual stimulus might be weaker at the boundary of these ranges for human visual system [21]. Then, we rendered in sRGB colour images from hyperspectral images to create ground-truth salient object binary images by labelling the boundaries of salient objects. In Fig.3, some example images are given from our hyperspectral dataset rendered in sRGB colour images with their respective ground truth binary images for salient objects.
Iii Experiments on dataset
In this section, we test the dataset with spectral saliency models presented in [3] and [4]. For quantitative evaluation of the salient object detection performances, Area Under Curve (AUC) metric is selected; AUC implementation of Borji et al. [22] (AUC-Borji) is used in our experiments. AUC is a commonly used metric for the comparison of salient object detection methods. The results are given in Table II.
We started out experiments by utilizing saliency computation from [3], in which the code of [3] demonstrates various usage of spectral data for saliency detection. First of all, as a baseline model, saliency maps from Itti et al. [13] were also computed for comparison in the code of [3]. Then, the work in [3] check spectral distances between each spatial region for saliency computation by using spectral Euclidean distance (SED) and spectral Angle distances (SAD). Also, in [3], colour opponency method in [13]
is replaced by spectral information rather than Red-Green and Blue-Yellow differences. To do compute saliency from spectral group (GS), spectral bands are divided into four groups (G1,G2,G3,G4), and then Euclidean distance between these vectors (G1-G3 and G2-G4) as colour opponency are calculated rather than single value colour component
[3]. In [3], orientation based salient features (OCM) are also adopted from [13]. As in [3], the combinations SED-OCM-GS and SED-OCM-SAD were also tested on our dataset. From the various spectral saliency approaches in [3], SED-OCM-SAD yielded best AUC performance by giving 0.8008.
As a more recent work, we also tested saliency from spectral gradient contrast (SGC) proposed by [4]. It should be noted that we implemented the model in [4] since the codes were not available yet. In [4], local region contrast is computed from the super-pixels, where super pixels are obtained by considering both spatial and spectral gradients. SGC [4] gives the best AUC performance on our dataset among the tested models by having 0.8205 AUC performance.
TABLE II: Evaluation of spectral salient object detection methods on our dataset
Iv Conclusion
In this work, we presented a collection of larger hyperspectral image dataset (60 images with respective salient object ground-truths) that can be used for salient object detection task. Then, we tested our hyperspectral data with some spectral saliency models from [3] and [4]. Regarding, salient object detection task, SGC [4] seems to be more robust compared to models in [3]
, probably, due to two main reasons; i) using region contrast may be less noisy than pixel-wise saliency, ii) spectral gradient may have higher invariance to illumination changes as stated in
[4]. However, despite being better than base line model [13], these initial spectral saliency results shows that there are still many things that can be proposed to improve spectral salient object detection performances since current AUC performances still does not seem to be at the level of state-of-the-art colour image based salient object detection methods. We hope that this dataset will help to improve research in this area.
Acknowledgement
This paper based on the results obtained from a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
References
[1] A. Chakrabarti and T. Zickler, Statistics of Real-World Hyperspectral Images
, in Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
[2] R. B. Smith, Introduction to Hyperspectral Imaging with TNTmips, 2012 online available: http://www.microimages.com
[3] J. Liang J. Zhou, X. Bai, Y. Qian, Salient object detection in hyperspectral images, IEEE Int. Conf. on Image Processing (ICIP), pp.2393-2397,2013.
[4] H. Yan, Y. Zhang, W. Wei, L. Zhang, Y. Li, Salient object detection in hyperspectral imagery using spectral gradient contrast, IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp.1560-1563,2016.
[5] D. Manolakis, R. Lockwood, T. Cooley, Hyperspectral Imaging Remote Sensing, Cambridge University Prss, Cambridge, United Kingdom, 2016.
[6] M. Borengasser, W. Hungate, R.Watkins, Hyperspectral Remote Sensing: Principles and Applications, CRC Press, Boca Raton FL, 2008.
[7] Y. Cao, J. Zhang, Q. Tian, L. Zhuo, Q. Zhou, Salient target detection in hyperspectral images using spectral saliency, IEEE ChinaSIP, pp.1086-1090, 2015.
[8] S. Le Moan, A. Mansouri, J. Hardeberg and Y. Voisin, Saliency in spectral images, in Proc. of the 17th Scandinavian Conference on Image Analysis, pp. 114-123, 2011.
[9] B. Arad and O. Ben-Shahar, Sparse recovery of hyperspectral signal from natural RGB images, in Proc. of the Europian Conference on Computer Vision (ECCV), pp.19-34, 2016.
[10] R. Kawakami, J. Wright, T. Yu-Wing, Y. Matsushita, M. Ben-Ezra, and K. Ikeuchi, High resolution hyperspectral imaging via matrix factorization, IEEE Conf. on Computer Vision and Pattern Recognition (IEEE CVPR), 2011.
[11] B. Arad and O. Ben-Shahar, Filter selection for hyperspectral estimation, in Proc. of the IEEE International Conference on Computer Vision (ICCV), pp.3153-3161, 2017.
[12] H. Kwon and Y. W. Tai, RGB-guided hyperspectral unsampling, in Proc. of the IEEE International Conference on Computer Vision (ICCV), pp.307-315, 2016.
[13] L. Itti, C. Koch, and E. Niebur, A model of saliency based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, no11, pp.1254-1259, 1998.
, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.35, no.1, pp.353-367, 2013.
[15] N. Imamoglu, Y. Fang, W. Yu, and W. Lin, A saliency detection model using low-level features based on Wavelet Transform, IEEE Transactions on Multimedia, vol.15, issue 1, pp.96-105, 2013.
[16] N. Imamoglu, C. Zhang, W. Shimoda, Y. Fang, and B. Shi, Saliency detection by forward and backward cues in deep CNNs, IEEE International Confetrence on Image Processing (ICIP), 2017.
[17] T. Liu, N. Zheng, X. Tang, and H.-Y. Shum, Learning to detect salient object, in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.353-367, 2007.
[21] D. H. Sliney, What is light? The visible spectrum and beyond, Eye (Nature), Vol.30, pp.222-229, 2016.
[22] A. Borji, H. R. Tavakoli, D. N. Sihite, and L. Itti Analysis of scores, datasets, and models in visual saliency prediction, IEEE International Conference on Computer Vision (ICCV), 2013.
share
