Robust deep learning-based semantic organ segmentation in hyperspectral images

by   Silvia Seidlitz, et al.

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases - consistently across modalities - with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean dice similarity coefficient (DSC) of 0.89 (standard deviation (SD) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 (SD 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information.


page 1

page 13

page 18

page 19

page 20

page 21

page 22

page 23


Semantic segmentation of surgical hyperspectral images under geometric domain shifts

Robust semantic segmentation of intraoperative image data could pave the...

Semantic segmentation of multispectral photoacoustic images using deep learning

Photoacoustic imaging has the potential to revolutionise healthcare due ...

Machine learning-based analysis of hyperspectral images for automated sepsis diagnosis

Sepsis is a leading cause of mortality and critical illness worldwide. W...

A Hyperspectral and RGB Dataset for Building Facade Segmentation

Hyperspectral Imaging (HSI) provides detailed spectral information and h...

Apple scab detection in orchards using deep learning on colour and multispectral images

Apple scab is a fungal disease caused by Venturia inaequalis. Disease is...

Intra-operative Brain Tumor Detection with Deep Learning-Optimized Hyperspectral Imaging

Surgery for gliomas (intrinsic brain tumors), especially when low-grade,...

Please sign up or login with your details

Forgot password? Click here to reset