DeepAI AI Chat
Log In Sign Up

Semantic segmentation of surgical hyperspectral images under geometric domain shifts

by   Jan Sellner, et al.

Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however, although common in real-world open surgeries due to variations in surgical procedures or situs occlusions, remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed "Organ Transplantation" that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and hyperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 latter's rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67 performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models will be made publicly available.


page 2

page 4

page 7

page 8

page 12


Robust deep learning-based semantic organ segmentation in hyperspectral images

Semantic image segmentation is an important prerequisite for context-awa...

m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks

Autonomous surgical procedures, in particular minimal invasive surgeries...

Augmentation Invariance and Adaptive Sampling in Semantic Segmentation of Agricultural Aerial Images

In this paper, we investigate the problem of Semantic Segmentation for a...

Learning Non-Unique Segmentation with Reward-Penalty Dice Loss

Semantic segmentation is one of the key problems in the field of compute...

Effective semantic segmentation in Cataract Surgery: What matters most?

Our work proposes neural network design choices that set the state-of-th...

Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding

Global and local relational reasoning enable scene understanding models ...

4D-OR: Semantic Scene Graphs for OR Domain Modeling

Surgical procedures are conducted in highly complex operating rooms (OR)...