Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room

by   Lena Maier-Hein, et al.

Image-based tracking of medical instruments is an integral part of many surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the methods proposed still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on robustness and generalization capabilities of the methods. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all frames in the videos as well as instance-wise segmentation masks for surgical instruments in more than 10,000 individual frames. The data has successfully been used to organize international competitions in the scope of the Endoscopic Vision Challenges (EndoVis) 2017 and 2019.


page 9

page 10

page 20

page 21

page 22

page 23

page 24

page 25


Robust Medical Instrument Segmentation Challenge 2019

Intraoperative tracking of laparoscopic instruments is often a prerequis...

U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument

Conventional therapy approaches limit surgeons' dexterity control due to...

Know your sensORs x2013 A Modality Study For Surgical Action Classification

The surgical operating room (OR) presents many opportunities for automat...

FaceOff: Anonymizing Videos in the Operating Rooms

Video capture in the surgical operating room (OR) is increasingly possib...

Coarse-to-fine Surgical Instrument Detection for Cataract Surgery Monitoring

The amount of surgical data, recorded during video-monitored surgeries, ...

Segmenting Medical Instruments in Minimally Invasive Surgeries using AttentionMask

Precisely locating and segmenting medical instruments in images of minim...

Situation Awareness for Automated Surgical Check-listing in AI-Assisted Operating Room

Nowadays, there are more surgical procedures that are being performed us...

Code Repositories


This repository contains video datasets that can be used for training coarse to fine-grained (phase, step and action) temporal classification tasks.

view repo