Remote Sensing Novel View Synthesis with Implicit Multiplane Representations

05/18/2022
by   Yongchang Wu, et al.
0

Novel view synthesis of remote sensing scenes is of great significance for scene visualization, human-computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for remote sensing images due to its high complexity, view sparsity and limited view-perspective variations. In this paper, we propose a novel remote sensing view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of remote sensing images, we represent the 3D space by combining implicit multiplane images (MPI) representation and deep neural networks. The 3D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multi-view input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as Implicit Multiplane Images (ImMPI). To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of remote sensing 3D scenes and proposed a neural network based Prior extractor to accelerate the optimization process. In addition, we propose a new dataset for remote sensing novel view synthesis with multi-view real-world google earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design. Our dataset and code can be found at https://github.com/wyc-Chang/ImMPI

READ FULL TEXT

page 1

page 3

page 7

page 8

page 9

page 10

page 11

research
01/02/2023

Credible Remote Sensing Scene Classification Using Evidential Fusion on Aerial-Ground Dual-view Images

Due to their ability to offer more comprehensive information than data f...
research
03/15/2023

Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation

The mainstream CNN-based remote sensing (RS) image semantic segmentation...
research
01/14/2023

Robust Remote Sensing Scene Classification with Multi-View Voting and Entropy Ranking

Deep convolutional neural networks have been widely used in scene classi...
research
07/04/2022

Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations

Neural Radiance Field (NeRF) regresses a neural parameterized scene by d...
research
03/31/2023

Efficient View Synthesis and 3D-based Multi-Frame Denoising with Multiplane Feature Representations

While current multi-frame restoration methods combine information from m...
research
06/05/2023

BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields

Neural rendering combines ideas from classical computer graphics and mac...
research
08/04/2020

Deep Multi Depth Panoramas for View Synthesis

We propose a learning-based approach for novel view synthesis for multi-...

Please sign up or login with your details

Forgot password? Click here to reset