Content-aware Warping for View Synthesis

01/22/2022
by   Mantang Guo, et al.
6

Existing image-based rendering methods usually adopt depth-based image warping operation to synthesize novel views. In this paper, we reason the essential limitations of the traditional warping operation to be the limited neighborhood and only distance-based interpolation weights. To this end, we propose content-aware warping, which adaptively learns the interpolation weights for pixels of a relatively large neighborhood from their contextual information via a lightweight neural network. Based on this learnable warping module, we propose a new end-to-end learning-based framework for novel view synthesis from two input source views, in which two additional modules, namely confidence-based blending and feature-assistant spatial refinement, are naturally proposed to handle the occlusion issue and capture the spatial correlation among pixels of the synthesized view, respectively. Besides, we also propose a weight-smoothness loss term to regularize the network. Experimental results on structured light field datasets with wide baselines and unstructured multi-view datasets show that the proposed method significantly outperforms state-of-the-art methods both quantitatively and visually. The source code will be publicly available at https://github.com/MantangGuo/CW4VS.

READ FULL TEXT

page 3

page 5

page 8

page 9

page 10

page 12

page 13

research
08/17/2021

Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines

In this paper, we tackle the problem of dense light field (LF) reconstru...
research
12/19/2018

Light Weight Color Image Warping with Inter-Channel Information

Image warping is a necessary step in many multimedia applications such a...
research
02/08/2018

Hole Filling with Multiple Reference Views in DIBR View Synthesis

Depth-image-based rendering (DIBR) oriented view synthesis has been wide...
research
01/07/2022

Auto-Weighted Layer Representation Based View Synthesis Distortion Estimation for 3-D Video Coding

Recently, various view synthesis distortion estimation models have been ...
research
08/20/2021

Out-of-boundary View Synthesis Towards Full-Frame Video Stabilization

Warping-based video stabilizers smooth camera trajectory by constraining...
research
08/19/2021

Real-time Image Enhancer via Learnable Spatial-aware 3D Lookup Tables

Recently, deep learning-based image enhancement algorithms achieved stat...
research
07/29/2022

End-to-end View Synthesis via NeRF Attention

In this paper, we present a simple seq2seq formulation for view synthesi...

Please sign up or login with your details

Forgot password? Click here to reset