Unsupervised Depth Completion with Calibrated Backprojection Layers

08/24/2021
by   Alex Wong, et al.
6

We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using the calibration matrix and a depth feature descriptor. The resulting 3D positional encoding is concatenated with the image descriptor and the previous layer output to yield the input to the next layer of the encoder. A decoder, exploiting skip-connections, produces a dense depth map. The resulting Calibrated Backprojection Network, or KBNet, is trained without supervision by minimizing the photometric reprojection error. KBNet imputes missing depth value based on the training set, rather than on generic regularization. We test KBNet on public depth completion benchmarks, where it outperforms the state of the art by 30 camera is used for training and testing. When the test camera is different, the improvement reaches 62 https://github.com/alexklwong/calibrated-backprojection-network.

READ FULL TEXT

page 4

page 7

page 8

page 13

page 14

page 15

page 16

page 17

research
01/28/2019

Dense Depth Posterior (DDP) from Single Image and Sparse Range

We present a deep learning system to infer the posterior distribution of...
research
10/08/2020

Refinement of Predicted Missing Parts Enhance Point Cloud Completion

Point cloud completion is the task of predicting complete geometry from ...
research
07/03/2020

ODE-CNN: Omnidirectional Depth Extension Networks

Omnidirectional 360 camera proliferates rapidly for autonomous robots si...
research
03/22/2018

CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks

3D LiDARs and 2D cameras are increasingly being used alongside each othe...
research
06/25/2020

Discontinuous and Smooth Depth Completion with Binary Anisotropic Diffusion Tensor

We propose an unsupervised real-time dense depth completion from a spars...
research
10/14/2017

An Adaptive Framework for Missing Depth Inference Using Joint Bilateral Filter

Depth imaging has largely focused on sensor and intrinsics properties. H...
research
06/06/2021

An Adaptive Framework for Learning Unsupervised Depth Completion

We present a method to infer a dense depth map from a color image and as...

Please sign up or login with your details

Forgot password? Click here to reset