High Speed and High Dynamic Range Video with an Event Camera

06/15/2019
by   Henri Rebecq, et al.
1

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20 while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. We also demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data.

READ FULL TEXT

page 10

page 14

page 18

page 19

page 20

page 21

page 22

page 24

research
04/17/2019

Events-to-Video: Bringing Modern Computer Vision to Event Cameras

Event cameras are novel sensors that report brightness changes in the fo...
research
04/24/2019

CED: Color Event Camera Dataset

Event cameras are novel, bio-inspired visual sensors, whose pixels outpu...
research
06/15/2022

E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations

Event cameras respond to brightness changes in the scene asynchronously ...
research
12/12/2021

Image Reconstruction from Events. Why learn it?

Traditional cameras measure image intensity. Event cameras, by contrast,...
research
03/25/2022

TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation

Recording fast motion in a high FPS (frame-per-second) requires expensiv...
research
04/07/2020

Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEs

Event-based cameras are novel, efficient sensors inspired by the human v...
research
12/10/2020

An Asynchronous Kalman Filter for Hybrid Event Cameras

We present an Asynchronous Kalman Filter (AKF) to reconstruct High Dynam...

Please sign up or login with your details

Forgot password? Click here to reset