Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation

03/27/2020
by   Shawn Mathew, et al.
10

Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models.

READ FULL TEXT

page 2

page 3

page 4

page 5

page 6

page 7

page 8

research
11/28/2019

An Efficient Multi-Domain Framework for Image-to-Image Translation

Existing approaches have been proposed to tackle unsupervised image-to-i...
research
02/13/2018

An Optimized Architecture for Unpaired Image-to-Image Translation

Unpaired Image-to-Image translation aims to convert the image from one d...
research
09/09/2019

Towards Learning a Self-inverse Network for Bidirectional Image-to-image Translation

The one-to-one mapping is necessary for many bidirectional image-to-imag...
research
08/26/2019

Mocycle-GAN: Unpaired Video-to-Video Translation

Unsupervised image-to-image translation is the task of translating an im...
research
03/31/2021

Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

Research in unpaired video translation has mainly focused on short-term ...
research
01/18/2021

Visualizing Missing Surfaces In Colonoscopy Videos using Shared Latent Space Representations

Optical colonoscopy (OC), the most prevalent colon cancer screening tool...

Please sign up or login with your details

Forgot password? Click here to reset