Deep Video Super-Resolution using HR Optical Flow Estimation

01/06/2020
by   Longguang Wang, et al.
5

Video super-resolution (SR) aims at generating a sequence of high-resolution (HR) frames with plausible and temporally consistent details from their low-resolution (LR) counterparts. The key challenge for video SR lies in the effective exploitation of temporal dependency between consecutive frames. Existing deep learning based methods commonly estimate optical flows between LR frames to provide temporal dependency. However, the resolution conflict between LR optical flows and HR outputs hinders the recovery of fine details. In this paper, we propose an end-to-end video SR network to super-resolve both optical flows and images. Optical flow SR from LR frames provides accurate temporal dependency and ultimately improves video SR performance. Specifically, we first propose an optical flow reconstruction network (OFRnet) to infer HR optical flows in a coarse-to-fine manner. Then, motion compensation is performed using HR optical flows to encode temporal dependency. Finally, compensated LR inputs are fed to a super-resolution network (SRnet) to generate SR results. Extensive experiments have been conducted to demonstrate the effectiveness of HR optical flows for SR performance improvement. Comparative results on the Vid4 and DAVIS-10 datasets show that our network achieves the state-of-the-art performance.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 4

page 5

page 7

page 8

page 11

page 12

page 14

09/23/2018

Learning for Video Super-Resolution through HR Optical Flow Estimation

Video super-resolution (SR) aims to generate a sequence of high-resoluti...
04/14/2022

Look Back and Forth: Video Super-Resolution with Explicit Temporal Difference Modeling

Temporal modeling is crucial for video super-resolution. Most of the vid...
03/13/2019

Two-Stream Oriented Video Super-Resolution for Action Recognition

We study the video super-resolution (SR) problem not for visual quality,...
07/23/2020

MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution

Video super-resolution (VSR) aims to utilize multiple low-resolution fra...
06/09/2021

Neural UpFlow: A Scene Flow Learning Approach to Increase the Apparent Resolution of Particle-Based Liquids

We present a novel up-resing technique for generating high-resolution li...
05/20/2022

Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration

How to properly model the inter-frame relation within the video sequence...
11/25/2019

Estimating People Flows to Better Count them in Crowded Scenes

State-of-the-art methods for counting people in crowded scenes rely on d...

Code Repositories

SOF-VSR

[ACCV 2018 & TIP 2020] SOF-VSR


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.