Optical Flow estimation with Event-based Cameras and Spiking Neural Networks

02/13/2023
by   Javier Cuadrado, et al.
0

Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or "spikes", when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates.

READ FULL TEXT

page 1

page 3

page 7

10/26/2017

Spiking Optical Flow for Event-based Sensors Using IBM's TrueNorth Neurosynaptic System

This paper describes a fully spike-based neural network for optical flow...
04/14/2023

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras

Optical flow provides information on relative motion that is an importan...
03/14/2020

Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

Event-based cameras display great potential for a variety of conditions ...
09/28/2021

StereoSpike: Depth Learning with a Spiking Neural Network

Depth estimation is an important computer vision task, useful in particu...
06/03/2021

Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

Neuromorphic sensing and computing hold a promise for highly energy-effi...
03/19/2023

Training a spiking neural network on an event-based label-free flow cytometry dataset

Imaging flow cytometry systems aim to analyze a huge number of cells or ...
03/13/2023

Dynamic Event-based Optical Identification and Communication

Optical identification is often done with spatial or temporal visual pat...

Code Repositories

OF_EV_SNN

Optical Flow estimation from Event cameras and Spiking Neural Networks


view repo

Please sign up or login with your details

Forgot password? Click here to reset