DeepAI AI Chat
Log In Sign Up

M-FUSE: Multi-frame Fusion for Scene Flow Estimation

07/12/2022
by   Lukas Mehl, et al.
University of Stuttgart
0

Recently, neural network for scene flow estimation show impressive results on automotive data such as the KITTI benchmark. However, despite of using sophisticated rigidity assumptions and parametrizations, such networks are typically limited to only two frame pairs which does not allow them to exploit temporal information. In our paper we address this shortcoming by proposing a novel multi-frame approach that considers an additional preceding stereo pair. To this end, we proceed in two steps: Firstly, building upon the recent RAFT-3D approach, we develop an advanced two-frame baseline by incorporating an improved stereo method. Secondly, and even more importantly, exploiting the specific modeling concepts of RAFT-3D, we propose a U-Net like architecture that performs a fusion of forward and backward flow estimates and hence allows to integrate temporal information on demand. Experiments on the KITTI benchmark do not only show that the advantages of the improved baseline and the temporal fusion approach complement each other, they also demonstrate that the computed scene flow is highly accurate. More precisely, our approach ranks second overall and first for the even more challenging foreground objects, in total outperforming the original RAFT-3D method by more than 16 at https://github.com/cv-stuttgart/M-FUSE.

READ FULL TEXT

page 3

page 4

page 6

07/05/2017

Fast Multi-frame Stereo Scene Flow with Motion Segmentation

We propose a new multi-frame method for efficiently computing scene flow...
11/24/2022

TemporalStereo: Efficient Spatial-Temporal Stereo Matching Network

We present TemporalStereo, a coarse-to-fine based online stereo matching...
11/03/2020

A Deep Temporal Fusion Framework for Scene Flow Using a Learnable Motion Model and Occlusions

Motion estimation is one of the core challenges in computer vision. With...
06/03/2018

ProFlow: Learning to Predict Optical Flow

Temporal coherence is a valuable source of information in the context of...
07/10/2018

SceneEDNet: A Deep Learning Approach for Scene Flow Estimation

Estimating scene flow in RGB-D videos is attracting much interest of the...
07/15/2022

Bi-PointFlowNet: Bidirectional Learning for Point Cloud Based Scene Flow Estimation

Scene flow estimation, which extracts point-wise motion between scenes, ...