Imaging spectroscopy devices can capture an information-rich representation of the scene comprised by tens or hundreds of wavelength-indexed bands. In contrast with their trichromatic (colour) counterparts, these images are composed of as many channels, each of these corresponding to a particular narrow-band segment of the electromagnetic spectrum . Thus, imaging spectroscopy has numerous applications in areas such as remote sensing [2, 3], disease diagnosis and image-guided surgery , food monitoring and safety , agriculture , archaeological conservation , astronomy 
and face recognition.
Recent advances in imaging spectroscopy have seen the development of sensors where the spectral filters are fully integrated into the complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) detectors. These are multi-spectral imaging devices which are single-shot and offer numerous advantages in terms of speed of acquisition and form-factor [10, 11]. However, one of the main drawbacks of these multispectral systems is the low raw spatial resolution per wavelength-indexed band in the image. Hence, super-resolving spectral images is crucial to achieving a much improved spatial resolution in these devices.
Note that, during recent years, there has been a steady improvement in the performance of example-based single image SR methods [12, 13, 14, 15]. This is partly due to the wide availability of various benchmark datasets for development and comparison. For example, the dataset introduced by Timofte et al. [16, 17], [18, 19, 20, 21], Urban100 , and DIV2K  are all widely available.
Similar to RGB or grey-scale super-resolution, recently example-based techniques for spectral image super-resolution have started to appear in the literature . However, in contrast to their RGB and grey-scale counterparts, multispectral/hyperspectral datasets suitable for the development of single image super-resolution are not as abundant or easily accessible. For example, the CNN-based method in  was developed by putting together three different hyperspectral datasets. The first of these, the CAVE  consists of only 35 hyperspectral and RGB pairs gathered in a laboratory setting and controlled lighting using a camera with tunable liquid crystal filters. Similarly, the second dataset from Harvard  contains fifty hyperspectral images captured with a time-multiplexed 31-channel camera with an integrated liquid crystal tunable filter. The third dataset is that in 
, which includes 25 hyperspectral images of outdoor urban and rural scenes also captured using a tunable liquid-crystal filter. Probably the largest spectral dataset to date with more than 250 31-channel spectral images is the one introduced with the NTIRE 2018 challenge on spectral reconstruction from RGB images.
Moreover, while the topic of spectral image super-resolution utilizing colour images, i.e., pan-sharpening, has been extensively studied [29, 30, 31] so as to develop efficient example-based super-resolution methods, stereo registered colour-spectral datasets are limited to small number of hyperspectral images. One of the very few examples is that of the datasets in 
, where the authors introduced a stereo RGB and near infrared (NIR) dataset of 477 images and propose a multispectral SIFT (MSIFT) method to register the images. However, the dataset is promoted in the context of scene recognition. In addition, the NIR images are comprised of only one wavelength-indexed band. Similarly, in, the authors introduce an RGB-NIR image dataset of approximately hours video with only one band dedicated to NIR images. The dataset was gathered in an urban setting by mounting the cameras on a vehicle.
In this paper we introduce a novel dataset of colour-multispectral images which we name StereoMSI. Unlike the above two RGB-NIR datasets, the dataset was primarily developed for the PIRM2018 spectral SR challenge111Refer to https://pirm2018.org/ for the spectral SR challenge and the dataset download links. 
and comprised 350 registered stereo RGB-spectral image pairs. The StereoMSI dataset is hence large enough to help develop deep learning spectral super-resolution methods. Moreover, it is, to the best of our knowledge, the first of its kind. As a result, the paper is organised as follows. We commence by introducing the dataset. We then present a number of image quality metrics over the dataset and the proposed splits for training, validation and testing. Then we present a brief review of the challenge and elaborate upon the results obtained by its participants. Finally, we discuss other potential applications of the dataset and conclude on the developments presented here.
2 StereoMSI Dataset
As mentioned above, here we propose the StereoMSI dataset. The dataset is a novel RGB-spectral stereo image dataset for benchmarking example-based single spectral image and example-based RGB-guided spectral image super-resolution methods. The dataset is developed for research purposes only.
2.1 Diversity and Resolution
The 350 stereo pair images were collected from a diverse range of scenery in the city of Canberra, the capital of Australia. The nature of the images ranges from open industrial to office environments and from deserts to rainforests. In Figures 2 and 3 we display validation images for the former and latter, respectively.
It is worth noting that, during acquisition time, we paid particular attention to the exposure time and image quality as the stereo pairs were captured using different cameras. One is an RGB XiQ camera model MQ022CG-CM and the other is a XiQ multispectral camera model MQ022HG-IM-SM4x4 covering the interval in the visible spectral range.
The original spectral images were processed and cropped to the resolution so as to allow the stereo RGB images to be resized to a resolution 2 times larger in each axis, that is . This is due to the fact that, in practice, the RGB camera used, based upon a CMOS image sensor, has a Bayer RGGB pattern whereas the IMEC spectral sensors have a pattern delivering 16 wavelength bands. Hence, the resolution of the RGB images in each axis is twice that of the spectral images. Figure 4 illustrates this resolution relationship between the two filter arrays on both cameras. When processing the images, no gamma correction was applied.
2.2 Structure and Splits
|Number of images||350||240||200||20||130||100||10||20|
|Spectral camera model||XiQ MQ022HG-IM-SM4x4|
|RGB camera model||XiQ MQ022CG-CM|
After collecting the StereoMSI 350 images, the two invalid wavelength-indexed bands on the IMEC sensor were removed. We then registered the images using Flownet2.0  and used MATLAB’s imresize222For more information on the imresize function, go to https://www.mathworks.com/help/images/ref/imresize.html function to obtain lower resolution versions of each image by downscaling them by factors of and
with nearest neighbour interpolation.
The dataset for Track 1 (single image super-resolution) consists of 240 different spectral images. The 240 images have been split into 200 for training, 20 for validation and 20 for testing with low resolution (HR) and high resolution (LR) on self explicatively named directories. The dataset for Track 2 (colour-guided spectral image super-resolution) consists of 120 randomly selected image stereo pairs, where one view is captured by the spectral imager and the other one by the colour camera. The images have been split into 100 pairs for training, 10 for validation and 10 for testing with HR, and LR on self explicatively named directories. All the images, for both tracks, are in band-sequential, 16 bit, ENVI standard file format. Table 1 summarized the dataset and camera properties explained above.
2.3 Bicubic Upsampling Metrics
To quantitatively assess our StereoMSI dataset, and to provide a baseline for future benchmarking, we have performed image upsampling by applying a bicubic kernel. Python’s imresize function from the scikit-image333https://scikit-image.org/ toolbox was used to perform bicubic upsampling. With the upsampled images in hand, we have then computed a number of image quality metrics so as to compare the performance of current and future example-based spectral super-resolution algorithms. To this end, we up-sampled the lower-resolution images in the dataset by and and compared against their HR reference counterparts.
For the sake of consistency, here we use the same metrics as those applied in the PIRM2018 spectral super-resolution challenge . This are the mean relative absolute error (MRAE) (introduced in ), the Spectral Information Divergence (SID), the per-band Mean Squared Error (MSE), the Average Per-Pixel Spectral Angle (APPSA), the average per-image Structural Similarity index (SSIM) and the mean per-image Peak Signal-to-Noise Ratio (PSNR). For more information on these metrics refer to the PIRM2018 spectral image super-resolution challenge report .
In Table 2, we show the image metric results for the whole 350 images comprising the StereoMSI dataset, and the testing images. We have included the testing split in the table since the testing imagery for both tracks is the same. Table 3, and 4 show the results for full dataset, training and validation splits of Track 1 and Track 2, respectively.
3 PIRM2018 Spectral Image Super-resolution Challenge
The PIRM2018 challenge has a twofold motivation. Firstly, the notion that, by using machine learning techniques, single image SR systems can be trained to obtain reliable multispectral super-resolved images at testing. Secondly, that by exploiting the higher resolution of the RGB images registered onto the spectral images, the performance of the algorithms can be further improved.
Track 1 focuses on to the problem of super-resolving the spatial resolution of spectral images given training image pairs, whereby one of these is an LR and the other one is an HR image, i.e. the ground truth reference image. The aim is hence to obtain spatially super-resolved spectral images making use of training imagery. Track 2, in the other hand, aims at obtaining spatially super-resolved spectral images making use of spectral-RGB stereo image pairs.
Each of the participating teams is expected to submit HR testing images which are to be evaluated with respect to several quantitative criteria concerning the fidelity of the reconstruction of the spectra in the super-resolved spectral images. The quantitative assessment of the fidelity of the images consists of the comparison of the restored multispectral images with their corresponding ground truth. For this, the challenge used the MRAE, the SID, the MSE, the APPSA, the SSIM and the mean PSNR. However, only MRAE and SID were used for ranking.
|Bicubic upsampling ()||0.21||0.00035||0.11||6353052||29.3||0.47|
In Table 5, we present the fidelity measurements for the testing images submitted by the challenge winners. Additionally, in Figure 5 we show sample super-resolved results for the two winners of the competition. For more details regarding the challenge, the super-resolution results obtained by other participants and the networks and algorithms used at the challenge, we would like to refer the interested reader to .
4 Discussion and Conclusions
In this paper, we have introduced the StereoMSI dataset comprising of 350 stereo spectral-colour image pairs. The dataset is a novel one which is specifically structured for multispectral super-resolution benchmarking. Although it was acquired with spectral image super-resolution in mind, it is quite general in nature. Having a ColorChecker present in every image, it can also be used for a number of other learning-based applications. Moreover, it also provides lower-resolution imagery and training, validation, and testing splits for both colour-guided and example-based learning applications. We have also presented a set of quality image metrics applied to the images when up-sampled using a bicubic kernel and, in doing so, provided a baseline based upon an image resizing approach widely used in the community. We have also provided a summary of both tracks in the PIRM2018 spectral image super-resolution challenge and shown the results obtained by the respective winners.
The PIRM2018 challenge was sponsored by CSIRO’s DATA61, Deakin University, ETH Zurich, HUAWEI, and MediaTek.
-  Robles-Kelly, A., Huynh, C.P.: Imaging spectroscopy for scene analysis. Springer Science & Business Media (2012)
-  Goetz, A.F.: Three decades of hyperspectral remote sensing of the earth: A personal view. Remote Sensing of Environment 113 (2009) S5–S16
-  Hasan, M., Jia, X., Robles-Kelly, A., Zhou, J., Pickering, M.R.: Multi-spectral remote sensing image registration via spatial relationship analysis on sift keypoints. In: Geoscience and Remote Sensing Symposium (IGARSS), 2010 IEEE International, IEEE (2010) 1011–1014
-  Lu, G., Fei, B.: Medical hyperspectral imaging: a review. Journal of biomedical optics 19(1) (2014) 010901
-  Feng, Y.Z., Sun, D.W.: Application of hyperspectral imaging in food safety inspection and control: a review. Critical reviews in food science and nutrition 52(11) (2012) 1039–1058
-  Elarab, M., Ticlavilca, A.M., Torres-Rua, A.F., Maslova, I., McKee, M.: Estimating chlorophyll with thermal and broadband multispectral high resolution imagery from an unmanned aerial system using relevance vector machines for precision agriculture. International Journal of Applied Earth Observation and Geoinformation 43 (2015) 32–42
-  Liang, H.: Advances in multispectral and hyperspectral imaging for archaeology and art conservation. Applied Physics A 106(2) (2012) 309–323
-  Bell, J.F., Wellington, D., Hardgrove, C., Godber, A., Rice, M.S., Johnson, J.R., Fraeman, A.: Multispectral imaging of mars from the mars science laboratory mastcam instruments: Spectral properties and mineralogic implications along the gale crater traverse. In: AAS/Division for Planetary Sciences Meeting Abstracts. Volume 48. (2016)
-  Xie, Z., Jiang, P., Zhang, S., Xiong, J.: Hyperspectral face recognition based on spatio-spectral fusion and local binary pattern. In: AOPC 2017: Optical Sensing and Imaging Technology and Applications. Volume 10462., International Society for Optics and Photonics (2017) 104620C
-  Wu, D., Sun, D.W.: Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment: A review—part i: Fundamentals. Innovative Food Science & Emerging Technologies 19 (2013) 1–14
-  Bigas, M., Cabruja, E., Forest, J., Salvi, J.: Review of cmos image sensors. Microelectronics journal 37(5) (2006) 433–451
Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L., Lim, B., Son,
S., Kim, H., Nah, S., Lee, K.M., et al.:
Ntire 2017 challenge on single image super-resolution: Methods and
In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, IEEE (2017) 1110–1121
-  Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 1646–1654
-  Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate superresolution. In: IEEE Conference on Computer Vision and Pattern Recognition. Volume 2. (2017) 5
-  Timofte, R., Gu, S., Wu, J., Van Gool, L., et al.: Ntire 2018 challenge on single image super-resolution: Methods and results. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. (June 2018)
-  Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on Computer Vision, Springer (2014) 111–126
-  Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision. (2013) 1920–1927
-  Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on. Volume 2., IEEE (2001) 416–423
-  Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE transactions on image processing 19(11) (2010) 2861–2873
-  Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International conference on curves and surfaces, Springer (2010) 711–730
-  Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. (2012)
-  Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 5197–5206
-  Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. Volume 3. (2017) 2
Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.:
Hyperspectral image super-resolution using deep convolutional neural network.Neurocomputing 266 (2017) 29–41
-  Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE transactions on image processing 19(9) (2010) 2241–2253
-  Chakrabarti, A., Zickler, T.: Statistics of real-world hyperspectral images. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE (2011) 193–200
-  Foster, D.H., Nascimento, S.M., Amano, K.: Information limits on neural identification of colored surfaces in natural scenes. Visual neuroscience 21(3) (2004) 331–336
-  Arad, B., Ben-Shahar, O., Timofte, R.: Ntire 2018 challenge on spectral reconstruction from rgb images. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. (June 2018)
-  Loncan, L., Almeida, L.B., Bioucas-Dias, J.M., Briottet, X., Chanussot, J., Dobigeon, N., Fabre, S., Liao, W., Licciardi, G.A., Simoes, M., et al.: Hyperspectral pansharpening: A review. arXiv preprint arXiv:1504.04531 (2015)
-  Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 3586–3594
-  Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.W., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE (2011) 2329–2336
-  Brown, M., Süsstrunk, S.: Multi-spectral sift for scene category recognition. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE (2011) 177–184
-  Zhi, T., Pires, B.R., Hebert, M., Narasimhan, S.G.: Deep material-aware cross-spectral stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2018) 1916–1925
-  Shoeiby, M., Robles-Kelly, A., Timofte, R., Zhou, R., Lahoud, F., Süsstrunk, S., Xiong, Z., Shi, Z., Chen, C., Liu, D., et al.: Pirm2018 challenge on spectral image super-resolution: methods and results. In: European Conference on Computer Vision, Springer (2018) 356–371
-  Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: IEEE conference on computer vision and pattern recognition (CVPR). Volume 2. (2017) 6
-  Lahoud, F., Zhou, R., Süsstrunk, S.: Multi-modal spectral image super-resolution. In: European Conference on Computer Vision, Springer (2018) 35–50
-  Shi, Z., Chen, C., Xiong, Z., Liu, D., Zha, Z.J., Wu, F.: Deep residual attention network for spectral image super-resolution. In: European Conference on Computer Vision Workshops (ECCVW). (2018)