A Single Frame and Multi-Frame Joint Network for 360-degree Panorama Video Super-Resolution

by   Hongying Liu, et al.

Spherical videos, also known as 360 (panorama) videos, can be viewed with various virtual reality devices such as computers and head-mounted displays. They attract large amount of interest since awesome immersion can be experienced when watching spherical videos. However, capturing, storing and transmitting high-resolution spherical videos are extremely expensive. In this paper, we propose a novel single frame and multi-frame joint network (SMFN) for recovering high-resolution spherical videos from low-resolution inputs. To take advantage of pixel-level inter-frame consistency, deformable convolutions are used to eliminate the motion difference between feature maps of the target frame and its neighboring frames. A mixed attention mechanism is devised to enhance the feature representation capability. The dual learning strategy is exerted to constrain the space of solution so that a better solution can be found. A novel loss function based on the weighted mean square error is proposed to emphasize on the super-resolution of the equatorial regions. This is the first attempt to settle the super-resolution of spherical videos, and we collect a novel dataset from the Internet, MiG Panorama Video, which includes 204 videos. Experimental results on 4 representative video clips demonstrate the efficacy of the proposed method. The dataset and code are available at https://github.com/lovepiano/SMFN_For_360VSR.



There are no comments yet.


page 1

page 3

page 4

page 5

page 8


Memory-Augmented Non-Local Attention for Video Super-Resolution

In this paper, we propose a novel video super-resolution method that aim...

Applying VertexShuffle Toward 360-Degree Video Super-Resolution on Focused-Icosahedral-Mesh

With the emerging of 360-degree image/video, augmented reality (AR) and ...

FISR: Deep Joint Frame Interpolation and Super-Resolution with A Multi-scale Temporal Loss

Super-resolution (SR) has been widely used to convert low-resolution leg...

Temporal Modulation Network for Controllable Space-Time Video Super-Resolution

Space-time video super-resolution (STVSR) aims to increase the spatial a...

Super-resolution of Omnidirectional Images Using Adversarial Learning

An omnidirectional image (ODI) enables viewers to look in every directio...

Large Motion Video Super-Resolution with Dual Subnet and Multi-Stage Communicated Upsampling

Video super-resolution (VSR) aims at restoring a video in low-resolution...

Multi-feature super-resolution network for cloth wrinkle synthesis

Existing physical cloth simulators suffer from expensive computation and...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.