Flow-Guided Sparse Transformer for Video Deblurring

01/06/2022
by   Jing Lin, et al.
0

Exploiting similar and sharper scene patches in spatio-temporal neighborhoods is critical for video deblurring. However, CNN-based methods show limitations in capturing long-range dependencies and modeling non-local self-similarity. In this paper, we propose a novel framework, Flow-Guided Sparse Transformer (FGST), for video deblurring. In FGST, we customize a self-attention module, Flow-Guided Sparse Window-based Multi-head Self-Attention (FGSW-MSA). For each query element on the blurry reference frame, FGSW-MSA enjoys the guidance of the estimated optical flow to globally sample spatially sparse yet highly related key elements corresponding to the same scene patch in neighboring frames. Besides, we present a Recurrent Embedding (RE) mechanism to transfer information from past frames and strengthen long-range temporal dependencies. Comprehensive experiments demonstrate that our proposed FGST outperforms state-of-the-art (SOTA) methods on both DVD and GOPRO datasets and even yields more visually pleasing results in real video deblurring. Code and models will be released to the public.

READ FULL TEXT

page 6

page 7

page 8

research
03/09/2022

Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

Many algorithms have been developed to solve the inverse problem of code...
research
07/30/2023

Video Frame Interpolation with Flow Transformer

Video frame interpolation has been actively studied with the development...
research
11/15/2021

Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-...
research
04/23/2023

TransFlow: Transformer as Flow Learner

Optical flow is an indispensable building block for various important co...
research
09/08/2023

CNN Injected Transformer for Image Exposure Correction

Capturing images with incorrect exposure settings fails to deliver a sat...
research
05/24/2019

SCRAM: Spatially Coherent Randomized Attention Maps

Attention mechanisms and non-local mean operations in general are key in...
research
01/24/2023

Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting

Transformers have been widely used for video processing owing to the mul...

Please sign up or login with your details

Forgot password? Click here to reset