Depth-Supervised NeRF for Multi-View RGB-D Operating Room Images

11/22/2022
by   Beerend G. A. Gerats, et al.
0

Neural Radiance Fields (NeRF) is a powerful novel technology for the reconstruction of 3D scenes from a set of images captured by static cameras. Renders of these reconstructions could play a role in virtual presence in the operating room (OR), e.g. for training purposes. In contrast to existing systems for virtual presence, NeRF can provide real instead of simulated surgeries. This work shows how NeRF can be used for view synthesis in the OR. A depth-supervised NeRF (DS-NeRF) is trained with three or five synchronised cameras that capture the surgical field in knee replacement surgery videos from the 4D-OR dataset. The algorithm is trained and evaluated for images in five distinct phases before and during the surgery. With qualitative analysis, we inspect views synthesised by a virtual camera that moves in 180 degrees around the surgical field. Additionally, we quantitatively inspect view synthesis from an unseen camera position in terms of PSNR, SSIM and LPIPS for the colour channels and in terms of MAE and error percentage for the estimated depth. DS-NeRF generates geometrically consistent views, also from interpolated camera positions. Views are generated from an unseen camera pose with an average PSNR of 17.8 and a depth estimation error of 2.10 missing of fine details, the synthesised views do not look photo-realistic. Our results show the potential of NeRF for view synthesis in the OR. Recent developments, such as NeRF for video synthesis and training speedups, require further exploration to reveal its full potential.

READ FULL TEXT

page 5

page 7

page 9

research
04/02/2020

Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera

This paper presents a new method to synthesize an image from arbitrary v...
research
11/23/2021

Distortion Reduction for Off-Center Perspective Projection of Panoramas

A single Panorama can be drawn perspectively without distortions in arbi...
research
12/06/2021

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

Neural radiance fields (NeRF) encode a scene into a neural representatio...
research
07/22/2022

InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images

We present a method for learning to generate unbounded flythrough videos...
research
06/21/2021

Moving in a 360 World: Synthesizing Panoramic Parallaxes from a Single Panorama

We present Omnidirectional Neural Radiance Fields (OmniNeRF), the first ...
research
07/26/2023

DisguisOR: Holistic Face Anonymization for the Operating Room

Purpose: Recent advances in Surgical Data Science (SDS) have contributed...
research
07/22/2023

Replay: Multi-modal Multi-view Acted Videos for Casual Holography

We introduce Replay, a collection of multi-view, multi-modal videos of h...

Please sign up or login with your details

Forgot password? Click here to reset