Applying VertexShuffle Toward 360-Degree Video Super-Resolution on Focused-Icosahedral-Mesh

06/21/2021
by   Na Li, et al.
0

With the emerging of 360-degree image/video, augmented reality (AR) and virtual reality (VR), the demand for analysing and processing spherical signals get tremendous increase. However, plenty of effort paid on planar signals that projected from spherical signals, which leading to some problems, e.g. waste of pixels, distortion. Recent advances in spherical CNN have opened up the possibility of directly analysing spherical signals. However, they pay attention to the full mesh which makes it infeasible to deal with situations in real-world application due to the extremely large bandwidth requirement. To address the bandwidth waste problem associated with 360-degree video streaming and save computation, we exploit Focused Icosahedral Mesh to represent a small area and construct matrices to rotate spherical content to the focused mesh area. We also proposed a novel VertexShuffle operation that can significantly improve both the performance and the efficiency compared to the original MeshConv Transpose operation introduced in UGSCNN. We further apply our proposed methods on super resolution model, which is the first to propose a spherical super-resolution model that directly operates on a mesh representation of spherical pixels of 360-degree data. To evaluate our model, we also collect a set of high-resolution 360-degree videos to generate a spherical image dataset. Our experiments indicate that our proposed spherical super-resolution model achieves significant benefits in terms of both performance and inference time compared to the baseline spherical super-resolution model that uses the simple MeshConv Transpose operation. In summary, our model achieves great super-resolution performance on 360-degree inputs, achieving 32.79 dB PSNR on average when super-resoluting 16x vertices on the mesh.

READ FULL TEXT
research
08/24/2020

A Single Frame and Multi-Frame Joint Network for 360-degree Panorama Video Super-Resolution

Spherical videos, also known as 360 (panorama) videos, can be viewed wit...
research
01/29/2022

Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback

Field of view (FoV) prediction is critical in 360-degree video multicast...
research
08/12/2019

Super-resolution of Omnidirectional Images Using Adversarial Learning

An omnidirectional image (ODI) enables viewers to look in every directio...
research
12/13/2021

SphereSR: 360° Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representation

The 360imaging has recently gained great attention; however, its angular...
research
01/11/2023

Learning Continuous Mesh Representation with Spherical Implicit Surface

As the most common representation for 3D shapes, mesh is often stored di...
research
07/23/2021

Generating N-point spherical configurations with low mesh ratios using spherical area coordinates

This short contribution presents a method for generating N-point spheric...
research
09/06/2022

Spherical Coordinates from Persistent Cohomology

We describe a method to obtain spherical parameterizations of arbitrary ...

Please sign up or login with your details

Forgot password? Click here to reset