DeepAI AI Chat
Log In Sign Up

Dense Continuous-Time Optical Flow from Events and Frames

by   Mathias Gehrig, et al.
Universität Zürich

We present a method for estimating dense continuous-time optical flow. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute per-pixel, continuous-time optical flow by additionally using events from an event camera. Events provide temporally fine-grained information about movement in image space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous-time via parameterized Bézier curves. To achieve this, we introduce multiple innovations to build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. The proposed approach outperforms existing image-based and event-based methods by 11.5 Finally, we introduce a novel synthetic dataset MultiFlow for pixel trajectory regression on which our method is currently the only successful approach.


page 1

page 10

page 13

page 22


Learning Dense and Continuous Optical Flow from an Event Camera

Event cameras such as DAVIS can simultaneously output high temporal reso...

Lightweight Event-based Optical Flow Estimation via Iterative Deblurring

Inspired by frame-based methods, state-of-the-art event-based optical fl...

Event-based Temporally Dense Optical Flow Estimation with Sequential Neural Networks

Prior works on event-based optical flow estimation have investigated sev...

DGC-Net: Dense Geometric Correspondence Network

This paper addresses the challenge of dense pixel correspondence estimat...

Simultaneous Optical Flow and Segmentation (SOFAS) using Dynamic Vision Sensor

We present an algorithm (SOFAS) to estimate the optical flow of events g...

Learning Optical Flow from Event Camera with Rendered Dataset

We study the problem of estimating optical flow from event cameras. One ...

Event-Based Feature Tracking in Continuous Time with Sliding Window Optimization

We propose a novel method for continuous-time feature tracking in event ...