Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation

07/14/2021
by   Lalith Sharan, et al.
0

The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance of the same surgical target structure. This can be viewed as a novel augmented reality approach, which we coined Hyperrealism in previous work. In this use case, it is of paramount importance to display objects like needles, sutures or instruments consistent in both domains while altering the style to a more tissue-like appearance. Segmentation of these objects would allow for a direct transfer, however, contouring of these, partly tiny and thin foreground objects is cumbersome and perhaps inaccurate. Instead, we propose to use landmark detection on the points when sutures pass into the tissue. This objective is directly incorporated into a CycleGAN framework by treating the performance of pre-trained detector models as an additional optimization goal. We show that a task defined on these sparse landmark labels improves consistency of synthesis by the generator network in both domains. Comparing a baseline CycleGAN architecture to our proposed extension (DetCycleGAN), mean precision (PPV) improved by +61.32, mean sensitivity (TPR) by +37.91, and mean F1 score by +0.4743. Furthermore, it could be shown that by dataset fusion, generated intra-operative images can be leveraged as additional training data for the detection network itself. The data is released within the scope of the AdaptOR MICCAI Challenge 2021 at https://adaptor2021.github.io/, and code at https://github.com/Cardio-AI/detcyclegan_pytorch.

READ FULL TEXT

page 1

page 6

page 9

page 10

research
12/04/2019

StarGAN v2: Diverse Image Synthesis for Multiple Domains

A good image-to-image translation model should learn a mapping between d...
research
11/16/2021

Point detection through multi-instance deep heatmap regression for sutures in endoscopy

Purpose: Mitral valve repair is a complex minimally invasive surgery of ...
research
06/10/2018

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Current `dry lab' surgical phantom simulators are a valuable tool for su...
research
08/22/2021

Graph2Pix: A Graph-Based Image to Image Translation Framework

In this paper, we propose a graph-based image-to-image translation frame...
research
08/26/2022

Region-guided CycleGANs for Stain Transfer in Whole Slide Images

In whole slide imaging, commonly used staining techniques based on hemat...
research
10/12/2021

Improved Heatmap-based Landmark Detection

Mitral valve repair is a very difficult operation, often requiring exper...
research
06/24/2019

Cross-Domain Conditional Generative Adversarial Networks for Stereoscopic Hyperrealism in Surgical Training

Phantoms for surgical training are able to mimic cutting and suturing pr...

Please sign up or login with your details

Forgot password? Click here to reset