Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events

04/14/2023
by   Yangguang Wang, et al.
0

Scene Dynamic Recovery (SDR) by inverting distorted Rolling Shutter (RS) images to an undistorted high frame-rate Global Shutter (GS) video is a severely ill-posed problem, particularly when prior knowledge about camera/object motions is unavailable. Commonly used artificial assumptions on motion linearity and data-specific characteristics, regarding the temporal dynamics information embedded in the RS scanlines, are prone to producing sub-optimal solutions in real-world scenarios. To address this challenge, we propose an event-based RS2GS framework within a self-supervised learning paradigm that leverages the extremely high temporal resolution of event cameras to provide accurate inter/intra-frame information. to leverage the event camera to provide inter/intra-frame information as the emitted events have an extremely high temporal resolution and learn an event-based RS2GS network within a self-supervised learning framework, where real-world events and RS images can be exploited to alleviate the performance degradation caused by the domain gap between the synthesized and real data. Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals, including the temporal transition and spatial translation. Exploring connections in terms of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the proposed E-IC, resulting in supervisions without ground-truth GS images. Extensive evaluations over synthetic and real datasets demonstrate that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios. The dataset and code are available at https://w3un.github.io/selfunroll/.

READ FULL TEXT

page 1

page 4

page 5

page 7

page 9

page 10

page 11

page 12

research
06/27/2023

Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames

This paper makes the first attempt to tackle the challenging task of rec...
research
05/31/2023

Self-supervised Learning to Bring Dual Reversed Rolling Shutter Images Alive

Modern consumer cameras usually employ the rolling shutter (RS) mechanis...
research
09/28/2021

Motion Deblurring with Real Events

In this paper, we propose an end-to-end learning framework for event-bas...
research
02/27/2023

Learning to Super-Resolve Blurry Images with Events

Super-Resolution from a single motion Blurred image (SRB) is a severely ...
research
08/11/2023

Generalizing Event-Based Motion Deblurring in Real-World Scenarios

Event-based motion deblurring has shown promising results by exploiting ...
research
04/05/2023

Recovering Continuous Scene Dynamics from A Single Blurry Image with Events

This paper aims at demystifying a single motion-blurred image with event...
research
10/08/2019

Learning event representations in image sequences by dynamic graph embedding

Recently, self-supervised learning has proved to be effective to learn r...

Please sign up or login with your details

Forgot password? Click here to reset