Multispectral Pan-sharpening via Dual-Channel Convolutional Network with Convolutional LSTM Based Hierarchical Spatial-Spectral Feature Fusion

07/20/2020
by   Dong Wang, et al.
0

Multispectral pan-sharpening aims at producing a high resolution (HR) multispectral (MS) image in both spatial and spectral domains by fusing a panchromatic (PAN) image and a corresponding MS image. In this paper, we propose a novel dual-channel network (DCNet) framework for MS pan-sharpening. In our DCNet, the dual-channel backbone involves a spatial channel to capture spatial information with a 2D CNN, and a spectral channel to extract spectral information with a 3D CNN. This heterogeneous 2D/3D CNN architecture can minimize causing spectral information distortion, which typically happens in conventional 2D CNN models. In order to fully integrate the spatial and spectral features captured from different levels, we introduce a multi-level fusion strategy. Specifically, a spatial-spectral CLSTM (S^2-CLSTM) module is proposed for fusing the hierarchical spatial and spectral features, which can effectively capture correlations among multi-level features. The S^2-CLSTM module attaches two fusion ways: the intra-level fusion via bi-directional lateral connections and inter-level fusion via the cell state in the S^2-CLSTM. Finally, the ideal HR-MS image is recovered by a reconstruction module. Extensive experiments have been conducted at both simulated lower scale and the original scale of real-world datasets. Compared with the state-of-the-art methods, the proposed DCNet achieves superior or competitive performance.

READ FULL TEXT

page 1

page 5

page 10

page 18

page 19

page 22

research
09/04/2018

Spatial-Spectral Fusion by Combining Deep Learning and Variation Model

In the field of spatial-spectral fusion, the model-based method and the ...
research
11/21/2018

A Deep Tree-Structured Fusion Model for Single Image Deraining

We propose a simple yet effective deep tree-structured fusion model base...
research
03/06/2022

PanFormer: a Transformer Based Model for Pan-sharpening

Pan-sharpening aims at producing a high-resolution (HR) multi-spectral (...
research
03/04/2022

HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening

Pansharpening aims to fuse a registered high-resolution panchromatic ima...
research
03/04/2022

HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging

The rapid development of deep learning provides a better solution for th...
research
12/04/2021

A Triple-Double Convolutional Neural Network for Panchromatic Sharpening

Pansharpening refers to the fusion of a panchromatic image with a high s...
research
01/27/2023

Mixed Attention Network for Hyperspectral Image Denoising

Hyperspectral image denoising is unique for the highly similar and corre...

Please sign up or login with your details

Forgot password? Click here to reset