Look Back and Forth: Video Super-Resolution with Explicit Temporal Difference Modeling

by   Takashi Isobe, et al.
Dalian University of Technology
Peking University
Beijing Kuaishou Technology Co.,Ltd.
The Hong Kong Polytechnic University

Temporal modeling is crucial for video super-resolution. Most of the video super-resolution methods adopt the optical flow or deformable convolution for explicitly motion compensation. However, such temporal modeling techniques increase the model complexity and might fail in case of occlusion or complex motion, resulting in serious distortion and artifacts. In this paper, we propose to explore the role of explicit temporal difference modeling in both LR and HR space. Instead of directly feeding consecutive frames into a VSR model, we propose to compute the temporal difference between frames and divide those pixels into two subsets according to the level of difference. They are separately processed with two branches of different receptive fields in order to better extract complementary information. To further enhance the super-resolution result, not only spatial residual features are extracted, but the difference between consecutive frames in high-frequency domain is also computed. It allows the model to exploit intermediate SR results in both future and past to refine the current SR output. The difference at different time steps could be cached such that information from further distance in time could be propagated to the current frame for refinement. Experiments on several video super-resolution benchmark datasets demonstrate the effectiveness of the proposed method and its favorable performance against state-of-the-art methods.


page 4

page 6

page 8


Deep Video Super-Resolution using HR Optical Flow Estimation

Video super-resolution (SR) aims at generating a sequence of high-resolu...

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

In this paper, we explore the space-time video super-resolution task, wh...

Local-Global Temporal Difference Learning for Satellite Video Super-Resolution

Optical-flow-based and kernel-based approaches have been widely explored...

Convolutional Video Steganography with Temporal Residual Modeling

Steganography represents the art of unobtrusively concealing a secrete m...

You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution

Spatial-Temporal Video Super-Resolution (ST-VSR) technology generates hi...

Frame and Feature-Context Video Super-Resolution

For video super-resolution, current state-of-the-art approaches either p...

Two-Stream Oriented Video Super-Resolution for Action Recognition

We study the video super-resolution (SR) problem not for visual quality,...

Please sign up or login with your details

Forgot password? Click here to reset