DeepAI
Log In Sign Up

Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion

08/15/2020
by   Igor Vasiljevic, et al.
15

Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model – usually the standard pinhole geometry – leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the camera model. Inspired by the geometric model of Grossberg and Nayar, we introduce Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise projection rays, approximating a wide range of cameras. NRS are fully differentiable and can be learned end-to-end from unlabeled raw videos. We demonstrate the use of NRS for self-supervised learning of visual odometry and depth estimation from raw videos obtained using a wide variety of camera systems, including pinhole, fisheye, and catadioptric.

READ FULL TEXT

page 1

page 6

page 9

page 10

page 11

12/06/2021

Self-Supervised Camera Self-Calibration from Video

Camera calibration is integral to robotics and computer vision algorithm...
10/04/2019

Two Stream Networks for Self-Supervised Ego-Motion Estimation

Learning depth and camera ego-motion from raw unlabeled RGB video stream...
11/13/2018

Self-Supervised Learning of Depth and Camera Motion from 360° Videos

As 360 cameras become prevalent in many autonomous systems (e.g., self-d...
10/13/2021

Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation

Estimating the motion of the camera together with the 3D structure of th...
06/08/2022

Learning Ego 3D Representation as Ray Tracing

A self-driving perception model aims to extract 3D semantic representati...
06/15/2022

Forecasting of depth and ego-motion with transformers and self-supervision

This paper addresses the problem of end-to-end self-supervised forecasti...