Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark

by   Martin Wagner, et al.

PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91 has been reported for phase recognition on an open data single-center dataset. In this work we investigated the generalizability of phase recognition algorithms in a multi-center setting including more difficult recognition tasks such as surgical action and surgical skill. METHODS: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 hours was created. Labels included annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 teams submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. RESULTS: F1-scores were achieved for phase recognition between 23.9 and 63.8 (n=5 teams). The average absolute error for skill assessment was 0.78 (n=1 team). CONCLUSION: Surgical workflow and skill analysis are promising technologies to support the surgical team, but are not solved yet, as shown by our comparison of algorithms. This novel benchmark can be used for comparable evaluation and validation of future work.


page 11

page 21

page 23

page 26

page 27

page 28


DeepPhase: Surgical Phase Recognition in CATARACTS Videos

Automated surgical workflow analysis and understanding can assist surgeo...

Surgical Phase and Instrument Recognition: How to identify appropriate Dataset Splits

Purpose: The development of machine learning models for surgical workflo...

Metrics Matter in Surgical Phase Recognition

Surgical phase recognition is a basic component for different context-aw...

PEg TRAnsfer Workflow recognition challenge report: Does multi-modal data improve recognition?

This paper presents the design and results of the "PEg TRAnsfert Workflo...

Offline identification of surgical deviations in laparoscopic rectopexy

Objective: A median of 14.4 during surgery and a third of them are preve...

MIcro-Surgical Anastomose Workflow recognition challenge report

The "MIcro-Surgical Anastomose Workflow recognition on training sessions...

The TUM LapChole dataset for the M2CAI 2016 workflow challenge

In this technical report we present our collected dataset of laparoscopi...

Please sign up or login with your details

Forgot password? Click here to reset