BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

by   Bowen Wen, et al.

We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation capturing both geometry and appearance. A dynamic pool of posed memory frames is automatically maintained to facilitate communication between these threads. Our approach handles challenging sequences with large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. We show results on HO3D, YCBInEOAT, and BEHAVE datasets, demonstrating that our method significantly outperforms existing approaches. Project page: https://bundlesdf.github.io


page 22

page 23

page 24

page 25

page 26

page 27

page 28

page 29


Tracking by 3D Model Estimation of Unknown Objects in Videos

Most model-free visual object tracking methods formulate the tracking ta...

A topological solution to object segmentation and tracking

The world is composed of objects, the ground, and the sky. Visual percep...

OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction

RGBD-based real-time dynamic 3D reconstruction suffers from inaccurate i...

A Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking

We propose an algorithm for real-time 6DOF pose tracking of rigid 3D obj...

Scene-level Tracking and Reconstruction without Object Priors

We present the first real-time system capable of tracking and reconstruc...

3D Reconstruction of Deformable Revolving Object under Heavy Hand Interaction

We reconstruct 3D deformable object through time, in the context of a li...

Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects

Tracking objects in 3D space and predicting their 6DoF pose is an essent...

Please sign up or login with your details

Forgot password? Click here to reset