Log In Sign Up

Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion

by   Igor Vasiljevic, et al.

Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model – usually the standard pinhole geometry – leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the camera model. Inspired by the geometric model of Grossberg and Nayar, we introduce Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise projection rays, approximating a wide range of cameras. NRS are fully differentiable and can be learned end-to-end from unlabeled raw videos. We demonstrate the use of NRS for self-supervised learning of visual odometry and depth estimation from raw videos obtained using a wide variety of camera systems, including pinhole, fisheye, and catadioptric.


page 1

page 6

page 9

page 10

page 11


Self-Supervised Camera Self-Calibration from Video

Camera calibration is integral to robotics and computer vision algorithm...

Two Stream Networks for Self-Supervised Ego-Motion Estimation

Learning depth and camera ego-motion from raw unlabeled RGB video stream...

Self-Supervised Learning of Depth and Camera Motion from 360° Videos

As 360 cameras become prevalent in many autonomous systems (e.g., self-d...

Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation

Estimating the motion of the camera together with the 3D structure of th...

Learning Ego 3D Representation as Ray Tracing

A self-driving perception model aims to extract 3D semantic representati...

Forecasting of depth and ego-motion with transformers and self-supervision

This paper addresses the problem of end-to-end self-supervised forecasti...