Real-time image-based instrument classification for laparoscopic surgery

by   Sebastian Bodenstedt, et al.

During laparoscopic surgery, context-aware assistance systems aim to alleviate some of the difficulties the surgeon faces. To ensure that the right information is provided at the right time, the current phase of the intervention has to be known. Real-time locating and classification the surgical tools currently in use are key components of both an activity-based phase recognition and assistance generation. In this paper, we present an image-based approach that detects and classifies tools during laparoscopic interventions in real-time. First, potential instrument bounding boxes are detected using a pixel-wise random forest segmentation. Each of these bounding boxes is then classified using a cascade of random forest. For this, multiple features, such as histograms over hue and saturation, gradients and SURF feature, are extracted from each detected bounding box. We evaluated our approach on five different videos from two different types of procedures. We distinguished between the four most common classes of instruments (LigaSure, atraumatic grasper, aspirator, clip applier) and background. Our method succesfully located up to 86 respectively. On manually provided bounding boxes, we achieve a instrument type recognition rate of up to 58 to 49 To our knowledge, this is the first approach that allows an image-based classification of surgical tools in a laparoscopic setting in real-time.


page 3

page 8


1st Place Solution for the UVO Challenge on Image-based Open-World Segmentation 2021

We describe our two-stage instance segmentation framework we use to comp...

Optimal Boxes: Boosting End-to-End Scene Text Recognition by Adjusting Annotated Bounding Boxes via Reinforcement Learning

Text detection and recognition are essential components of a modern OCR ...

Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance

Intra-operative anticipation of instrument usage is a necessary componen...

Localizing Firearm Carriers by Identifying Human-Object Pairs

Visual identification of gunmen in a crowd is a challenging problem, tha...

Multi-label Classification of Surgical Tools with Convolutional Neural Networks

Automatic tool detection from surgical imagery has a multitude of useful...

Fast and Regularized Reconstruction of Building Façades from Street-View Images using Binary Integer Programming

Regularized arrangement of primitives on building façades to aligned loc...

A Baseline Statistical Method For Robust User-Assisted Multiple Segmentation

Recently, several image segmentation methods that welcome and leverage d...