Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

11/27/2022
by   Ruofeng Wei, et al.
0

Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-based surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geometric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demonstrate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.

READ FULL TEXT

page 1

page 2

page 4

page 5

research
06/02/2018

Monocular Depth Estimation with Augmented Ordinal Depth Relationships

Most existing algorithms for depth estimation from single monocular imag...
research
10/08/2021

Stereo Dense Scene Reconstruction and Accurate Laparoscope Localization for Learning-Based Navigation in Robot-Assisted Surgery

The computation of anatomical information and laparoscope position is a ...
research
08/23/2022

Depth Map Decomposition for Monocular Depth Estimation

We propose a novel algorithm for monocular depth estimation that decompo...
research
11/23/2020

Data-driven Holistic Framework for Automated Laparoscope Optimal View Control with Learning-based Depth Perception

Laparoscopic Field of View (FOV) control is one of the most fundamental ...
research
03/04/2022

3D endoscopic depth estimation using 3D surface-aware constraints

Robotic-assisted surgery allows surgeons to conduct precise surgical ope...
research
12/21/2020

Monocular Depth Parameterizing Networks

Monocular depth estimation is a highly challenging problem that is often...
research
09/14/2023

An Explicit Method for Fast Monocular Depth Recovery in Corridor Environments

Monocular cameras are extensively employed in indoor robotics, but their...

Please sign up or login with your details

Forgot password? Click here to reset