Towards Unsupervised Learning for Instrument Segmentation in Robotic Surgery with Cycle-Consistent Adversarial Networks

07/09/2020
by   Daniil Pakhomov, et al.
13

Surgical tool segmentation in endoscopic images is an important problem: it is a crucial step towards full instrument pose estimation and it is used for integration of pre- and intra-operative images into the endoscopic view. While many recent approaches based on convolutional neural networks have shown great results, a key barrier to progress lies in the acquisition of a large number of manually-annotated images which is necessary for an algorithm to generalize and work well in diverse surgical scenarios. Unlike the surgical image data itself, annotations are difficult to acquire and may be of variable quality. On the other hand, synthetic annotations can be automatically generated by using forward kinematic model of the robot and CAD models of tools by projecting them onto an image plane. Unfortunately, this model is very inaccurate and cannot be used for supervised learning of image segmentation models. Since generated annotations will not directly correspond to endoscopic images due to errors, we formulate the problem as an unpaired image-to-image translation where the goal is to learn the mapping between an input endoscopic image and a corresponding annotation using an adversarial model. Our approach allows to train image segmentation models without the need to acquire expensive annotations and can potentially exploit large unlabeled endoscopic image collection outside the annotated distributions of image/annotation data. We test our proposed method on Endovis 2017 challenge dataset and show that it is competitive with supervised segmentation methods.

READ FULL TEXT

page 1

page 3

page 4

research
02/13/2019

Self-Supervised Surgical Tool Segmentation using Kinematic Information

Surgical tool segmentation in endoscopic images is the first step toward...
research
07/17/2020

Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery

Semantic tool segmentation in surgical videos is important for surgical ...
research
08/09/2023

SegMatch: A semi-supervised learning method for surgical instrument segmentation

Surgical instrument segmentation is recognised as a key enabler to provi...
research
02/16/2022

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

Automatic surgical instrument segmentation of endoscopic images is a cru...
research
08/27/2020

Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion

Surgical instrument segmentation is a key component in developing contex...
research
06/30/2020

Hand-drawn Symbol Recognition of Surgical Flowsheet Graphs with Deep Image Segmentation

Perioperative data are essential to investigating the causes of adverse ...
research
02/28/2021

A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images

3-D pose estimation of instruments is a crucial step towards automatic s...

Please sign up or login with your details

Forgot password? Click here to reset