Semantic segmentation of surgical hyperspectral images under geometric domain shifts

03/20/2023
by   Jan Sellner, et al.
18

Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however, although common in real-world open surgeries due to variations in surgical procedures or situs occlusions, remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed "Organ Transplantation" that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and hyperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 latter's rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67 performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models will be made publicly available.

READ FULL TEXT

page 2

page 4

page 7

page 8

page 12

research
11/09/2021

Robust deep learning-based semantic organ segmentation in hyperspectral images

Semantic image segmentation is an important prerequisite for context-awa...
research
08/23/2020

m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks

Autonomous surgical procedures, in particular minimal invasive surgeries...
research
04/17/2022

Augmentation Invariance and Adaptive Sampling in Semantic Segmentation of Agricultural Aerial Images

In this paper, we investigate the problem of Semantic Segmentation for a...
research
09/23/2020

Learning Non-Unique Segmentation with Reward-Penalty Dice Loss

Semantic segmentation is one of the key problems in the field of compute...
research
08/13/2021

Effective semantic segmentation in Cataract Surgery: What matters most?

Our work proposes neural network design choices that set the state-of-th...
research
09/11/2018

Normalization in Training Deep Convolutional Neural Networks for 2D Bio-medical Semantic Segmentation

2D bio-medical semantic segmentation is important for surgical robotic v...
research
03/22/2022

4D-OR: Semantic Scene Graphs for OR Domain Modeling

Surgical procedures are conducted in highly complex operating rooms (OR)...

Please sign up or login with your details

Forgot password? Click here to reset