Stereo Video Reconstruction Without Explicit Depth Maps for Endoscopic Surgery

09/16/2021
by   Annika Brundyn, et al.
29

We introduce the task of stereo video reconstruction or, equivalently, 2D-to-3D video conversion for minimally invasive surgical video. We design and implement a series of end-to-end U-Net-based solutions for this task by varying the input (single frame vs. multiple consecutive frames), loss function (MSE, MAE, or perceptual losses), and network architecture. We evaluate these solutions by surveying ten experts - surgeons who routinely perform endoscopic surgery. We run two separate reader studies: one evaluating individual frames and the other evaluating fully reconstructed 3D video played on a VR headset. In the first reader study, a variant of the U-Net that takes as input multiple consecutive video frames and outputs the missing view performs best. We draw two conclusions from this outcome. First, motion information coming from multiple past frames is crucial in recreating stereo vision. Second, the proposed U-Net variant can indeed exploit such motion information for solving this task. The result from the second study further confirms the effectiveness of the proposed U-Net variant. The surgeons reported that they could successfully perceive depth from the reconstructed 3D video clips. They also expressed a clear preference for the reconstructed 3D video over the original 2D video. These two reader studies strongly support the usefulness of the proposed task of stereo reconstruction for minimally invasive surgical video and indicate that deep learning is a promising approach to this task. Finally, we identify two automatic metrics, LPIPS and DISTS, that are strongly correlated with expert judgement and that could serve as proxies for the latter in future studies.

READ FULL TEXT

page 8

page 15

research
09/12/2017

End-to-End United Video Dehazing and Detection

The recent development of CNN-based image dehazing has revealed the effe...
research
12/11/2018

DeepV2D: Video to Depth with Differentiable Structure from Motion

We propose DeepV2D, an end-to-end differentiable deep learning architect...
research
02/26/2022

Accurate Human Body Reconstruction for Volumetric Video

In this work, we enhance a professional end-to-end volumetric video prod...
research
09/29/2020

Neural Alignment for Face De-pixelization

We present a simple method to reconstruct a high-resolution video from a...
research
04/18/2023

Saliency-aware Stereoscopic Video Retargeting

Stereo video retargeting aims to resize an image to a desired aspect rat...
research
03/16/2023

Shifted-Windows Transformers for the Detection of Cerebral Aneurysms in Microsurgery

Purpose: Microsurgical Aneurysm Clipping Surgery (MACS) carries a high r...
research
09/29/2022

Bounded Future MS-TCN++ for surgical gesture recognition

In recent times there is a growing development of video based applicatio...

Please sign up or login with your details

Forgot password? Click here to reset