Dense Continuous-Time Optical Flow from Events and Frames

by   Mathias Gehrig, et al.

We present a method for estimating dense continuous-time optical flow. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute per-pixel, continuous-time optical flow by additionally using events from an event camera. Events provide temporally fine-grained information about movement in image space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous-time via parameterized Bézier curves. To achieve this, we introduce multiple innovations to build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. The proposed approach outperforms existing image-based and event-based methods by 11.5 Finally, we introduce a novel synthetic dataset MultiFlow for pixel trajectory regression on which our method is currently the only successful approach.


page 1

page 10

page 13

page 22


Learning Dense and Continuous Optical Flow from an Event Camera

Event cameras such as DAVIS can simultaneously output high temporal reso...

Lightweight Event-based Optical Flow Estimation via Iterative Deblurring

Inspired by frame-based methods, state-of-the-art event-based optical fl...

Event-based Temporally Dense Optical Flow Estimation with Sequential Neural Networks

Prior works on event-based optical flow estimation have investigated sev...

DGC-Net: Dense Geometric Correspondence Network

This paper addresses the challenge of dense pixel correspondence estimat...

Simultaneous Optical Flow and Segmentation (SOFAS) using Dynamic Vision Sensor

We present an algorithm (SOFAS) to estimate the optical flow of events g...

Hallucinating Dense Optical Flow from Sparse Lidar for Autonomous Vehicles

In this paper we propose a novel approach to estimate dense optical flow...

Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency

The existing approaches for salient motion segmentation are unable to ex...

Please sign up or login with your details

Forgot password? Click here to reset