EventGAN: Leveraging Large Scale Image Datasets for Event Cameras

by   Alex Zihao Zhu, et al.

Event cameras provide a number of benefits over traditional cameras, such as the ability to track incredibly fast motions, high dynamic range, and low power consumption. However, their application into computer vision problems, many of which are primarily dominated by deep learning solutions, has been limited by the lack of labeled training data for events. In this work, we propose a method which leverages the existing labeled data for images by simulating events from a pair of temporal image frames, using a convolutional neural network. We train this network on pairs of images and events, using an adversarial discriminator loss and a pair of cycle consistency losses. The cycle consistency losses utilize a pair of pre-trained self-supervised networks which perform optical flow estimation and image reconstruction from events, and constrain our network to generate events which result in accurate outputs from both of these networks. Trained fully end to end, our network learns a generative model for events from images without the need for accurate modeling of the motion in the scene, exhibited by modeling based methods, while also implicitly modeling event noise. Using this simulator, we train a pair of downstream networks on object detection and 2D human pose estimation from events, using simulated data from large scale image datasets, and demonstrate the networks' abilities to generalize to datasets with real events.


page 3

page 4

page 6

page 7

page 8


EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

Event-based cameras have shown great promise in a variety of situations ...

Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy

Event cameras are novel vision sensors that sample, in an asynchronous f...

Masked Event Modeling: Self-Supervised Pretraining for Event Cameras

Event cameras offer the capacity to asynchronously capture brightness ch...

Learning to Detect Objects with a 1 Megapixel Event Camera

Event cameras encode visual information with high temporal precision, lo...

EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning

Event cameras sense intensity changes and have many advantages over conv...

Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation

Event cameras are novel sensors with outstanding properties such as high...

Mitosis Detection from Partial Annotation by Dataset Generation via Frame-Order Flipping

Detection of mitosis events plays an important role in biomedical resear...

Please sign up or login with your details

Forgot password? Click here to reset