Learning Optical Flow, Depth, and Scene Flow without Real-World Labels

03/28/2022
by   Vitor Guizilini, et al.
13

Self-supervised monocular depth estimation enables robots to learn 3D perception from raw video streams. This scalable approach leverages projective geometry and ego-motion to learn via view synthesis, assuming the world is mostly static. Dynamic scenes, which are common in autonomous driving and human-robot interaction, violate this assumption. Therefore, they require modeling dynamic objects explicitly, for instance via estimating pixel-wise 3D motion, i.e. scene flow. However, the simultaneous self-supervised learning of depth and scene flow is ill-posed, as there are infinitely many combinations that result in the same 3D point. In this paper we propose DRAFT, a new method capable of jointly learning depth, optical flow, and scene flow by combining synthetic data with geometric self-supervision. Building upon the RAFT architecture, we learn optical flow as an intermediate task to bootstrap depth and scene flow learning via triangulation. Our algorithm also leverages temporal and geometric consistency losses across tasks to improve multi-task learning. Our DRAFT architecture simultaneously establishes a new state of the art in all three tasks in the self-supervised monocular setting on the standard KITTI benchmark. Project page: https://sites.google.com/tri.global/draft.

READ FULL TEXT

page 1

page 3

page 7

research
12/09/2019

Self-supervised Object Motion and Depth Estimation from Video

We present a self-supervised learning framework to estimate the individu...
research
04/05/2023

DEFLOW: Self-supervised 3D Motion Estimation of Debris Flow

Existing work on scene flow estimation focuses on autonomous driving and...
research
03/31/2020

Distilled Semantics for Comprehensive Scene Understanding from Videos

Whole understanding of the surroundings is paramount to autonomous syste...
research
04/21/2023

FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving

Predicting accurate depth with monocular images is important for low-cos...
research
06/02/2023

The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation

Denoising diffusion probabilistic models have transformed image generati...
research
04/14/2022

Imposing Consistency for Optical Flow Estimation

Imposing consistency through proxy tasks has been shown to enhance data-...
research
08/25/2022

A Compacted Structure for Cross-domain learning on Monocular Depth and Flow Estimation

Accurate motion and depth recovery is important for many robot vision ta...

Please sign up or login with your details

Forgot password? Click here to reset