EventGAN: Leveraging Large Scale Image Datasets for Event Cameras

12/03/2019
by   Alex Zihao Zhu, et al.
16

Event cameras provide a number of benefits over traditional cameras, such as the ability to track incredibly fast motions, high dynamic range, and low power consumption. However, their application into computer vision problems, many of which are primarily dominated by deep learning solutions, has been limited by the lack of labeled training data for events. In this work, we propose a method which leverages the existing labeled data for images by simulating events from a pair of temporal image frames, using a convolutional neural network. We train this network on pairs of images and events, using an adversarial discriminator loss and a pair of cycle consistency losses. The cycle consistency losses utilize a pair of pre-trained self-supervised networks which perform optical flow estimation and image reconstruction from events, and constrain our network to generate events which result in accurate outputs from both of these networks. Trained fully end to end, our network learns a generative model for events from images without the need for accurate modeling of the motion in the scene, exhibited by modeling based methods, while also implicitly modeling event noise. Using this simulator, we train a pair of downstream networks on object detection and 2D human pose estimation from events, using simulated data from large scale image datasets, and demonstrate the networks' abilities to generalize to datasets with real events.

READ FULL TEXT

page 3

page 4

page 6

page 7

page 8

research
02/19/2018

EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

Event-based cameras have shown great promise in a variety of situations ...
research
09/17/2020

Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy

Event cameras are novel vision sensors that sample, in an asynchronous f...
research
12/20/2022

Masked Event Modeling: Self-Supervised Pretraining for Event Cameras

Event cameras offer the capacity to asynchronously capture brightness ch...
research
09/28/2020

Learning to Detect Objects with a 1 Megapixel Event Camera

Event cameras encode visual information with high temporal precision, lo...
research
03/17/2020

EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning

Event cameras sense intensity changes and have many advantages over conv...
research
09/06/2021

Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation

Event cameras are novel sensors with outstanding properties such as high...
research
07/09/2023

Mitosis Detection from Partial Annotation by Dataset Generation via Frame-Order Flipping

Detection of mitosis events plays an important role in biomedical resear...

Please sign up or login with your details

Forgot password? Click here to reset