DeepAI AI Chat
Log In Sign Up

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

by   Jiaming Sun, et al.

We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This design allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time.


page 3

page 4

page 8


PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos

We present PlanarRecon – a novel framework for globally coherent detecti...

Learning Human Mesh Recovery in 3D Scenes

We present a novel method for recovering the absolute pose and shape of ...

TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo

In this paper, we present TANDEM a real-time monocular tracking and dens...

MonoPerfCap: Human Performance Capture from Monocular Video

We present the first marker-less approach for temporally coherent 3D per...

ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction

One of the key elements of reconstructing a 3D mesh from a monocular vid...

SST: Real-time End-to-end Monocular 3D Reconstruction via Sparse Spatial-Temporal Guidance

Real-time monocular 3D reconstruction is a challenging problem that rema...

Incremental Dense Reconstruction from Monocular Video with Guided Sparse Feature Volume Fusion

Incrementally recovering 3D dense structures from monocular videos is of...

Code Repositories


Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral

view repo