Event-based Synthetic Aperture Imaging with a Hybrid Network

03/03/2021
by   Xiang Zhang, et al.
0

Synthetic aperture imaging (SAI) is able to achieve the see through effect by blurring out the off-focus foreground occlusions and reconstructing the in-focus occluded targets from multi-view images. However, very dense occlusions and extreme lighting conditions may bring significant disturbances to SAI based on conventional frame-based cameras, leading to performance degeneration. To address these problems, we propose a novel SAI system based on the event camera which can produce asynchronous events with extremely low latency and high dynamic range. Thus, it can eliminate the interference of dense occlusions by measuring with almost continuous views, and simultaneously tackle the over/under exposure problems. To reconstruct the occluded targets, we propose a hybrid encoder-decoder network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs). In the hybrid network, the spatio-temporal information of the collected events is first encoded by SNN layers, and then transformed to the visual image of the occluded targets by a style-transfer CNN decoder. Through experiments, the proposed method shows remarkable performance in dealing with very dense occlusions and extreme lighting conditions, and high quality visual images can be reconstructed using pure event data.

READ FULL TEXT

page 1

page 3

page 7

page 8

page 10

page 11

page 12

research
12/05/2022

Learning to See Through with Events

Although synthetic aperture imaging (SAI) can achieve the seeing-through...
research
12/10/2019

DeOccNet: Learning to See Through Foreground Occlusions in Light Fields

Background objects occluded in some views of a light field (LF) camera c...
research
07/28/2023

Seeing Behind Dynamic Occlusions with Event Cameras

Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, ...
research
06/05/2023

Best of Both Worlds: Hybrid SNN-ANN Architecture for Event-based Optical Flow Estimation

Event-based cameras offer a low-power alternative to frame-based cameras...
research
06/24/2022

Ev-NeRF: Event Based Neural Radiance Field

We present Ev-NeRF, a Neural Radiance Field derived from event data. Whi...
research
07/03/2023

Unveiling the Potential of Spike Streams for Foreground Occlusion Removal from Densely Continuous Views

The extraction of a clean background image by removing foreground occlus...
research
12/11/2019

PuckNet: Estimating hockey puck location from broadcast video

Puck location in ice hockey is essential for hockey analysts for determi...

Please sign up or login with your details

Forgot password? Click here to reset