End-to-End Learning of Video Super-Resolution with Motion Compensation

07/03/2017
by   Osama Makansi, et al.
0

Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.

READ FULL TEXT

page 7

page 10

page 16

research
03/20/2022

Optical Flow for Video Super-Resolution: A Survey

Video super-resolution is currently one of the most active research topi...
research
09/03/2018

Optical Flow Super-Resolution Based on Image Guidence Using Convolutional Neural Network

The convolutional neural network model for optical flow estimation usual...
research
11/24/2017

Video Enhancement with Task-Oriented Flow

Many video processing algorithms rely on optical flow to register differ...
research
04/09/2022

HSTR-Net: High Spatio-Temporal Resolution Video Generation For Wide Area Surveillance

Wide area surveillance has many applications and tracking of objects und...
research
07/27/2018

CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping

The Reference-based Super-resolution (RefSR) super-resolves a low-resolu...
research
09/23/2018

Learning for Video Super-Resolution through HR Optical Flow Estimation

Video super-resolution (SR) aims to generate a sequence of high-resoluti...
research
04/13/2018

Deep Motion Boundary Detection

Motion boundary detection is a crucial yet challenging problem. Prior me...

Please sign up or login with your details

Forgot password? Click here to reset