Exploiting Event Cameras by Using a Network Grafting Algorithm

03/24/2020
by   Yuhuang Hu, et al.
6

Novel vision sensors such as event cameras provide information that is not available from conventional intensity cameras. An obstacle to using these sensors with current powerful deep neural networks is the lack of large labeled training datasets. This paper proposes a Network Grafting Algorithm (NGA), where a new front end network driven by unconventional visual inputs replaces the front end network of a pretrained deep network that processes intensity frames. The self-supervised training uses only synchronously-recorded intensity frames and novel sensor data to maximize feature similarity between the pretrained network and the grafted network. We show that the enhanced grafted network reaches comparable average precision (AP_50) scores to the pretrained network on an object detection task using an event camera dataset, with no increase in inference costs. The grafted front end has only 5–8 the total parameters and can be trained in a few hours on a single GPU equivalent to 5 detector from labeled data. NGA allows these new vision sensors to capitalize on previously pretrained powerful deep models, saving on training cost.

READ FULL TEXT

page 2

page 8

page 10

research
12/06/2019

Video to Events: Bringing Modern Computer Vision Closer to Event Cameras

Event cameras are novel sensors that output brightness changes in the fo...
research
12/20/2022

Masked Event Modeling: Self-Supervised Pretraining for Event Cameras

Event cameras offer the capacity to asynchronously capture brightness ch...
research
07/27/2022

Traffic Sign Detection With Event Cameras and DCNN

In recent years, event cameras (DVS - Dynamic Vision Sensors) have been ...
research
02/07/2022

T-NGA: Temporal Network Grafting Algorithm for Learning to Process Spiking Audio Sensor Events

Spiking silicon cochlea sensors encode sound as an asynchronous stream o...
research
09/06/2021

Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation

Event cameras are novel sensors with outstanding properties such as high...
research
12/17/2021

Enhanced Frame and Event-Based Simulator and Event-Based Video Interpolation Network

Fast neuromorphic event-based vision sensors (Dynamic Vision Sensor, DVS...
research
06/30/2016

Steering a Predator Robot using a Mixed Frame/Event-Driven Convolutional Neural Network

This paper describes the application of a Convolutional Neural Network (...

Please sign up or login with your details

Forgot password? Click here to reset