Unsupervised Pansharpening Based on Self-Attention Mechanism

06/16/2020
by   Ying Qu, et al.
0

Pansharpening is to fuse a multispectral image (MSI) of low-spatial-resolution (LR) but rich spectral characteristics with a panchromatic image (PAN) of high-spatial-resolution (HR) but poor spectral characteristics. Traditional methods usually inject the extracted high-frequency details from PAN into the up-sampled MSI. Recent deep learning endeavors are mostly supervised assuming the HR MSI is available, which is unrealistic especially for satellite images. Nonetheless, these methods could not fully exploit the rich spectral characteristics in the MSI. Due to the wide existence of mixed pixels in satellite images where each pixel tends to cover more than one constituent material, pansharpening at the subpixel level becomes essential. In this paper, we propose an unsupervised pansharpening (UP) method in a deep-learning framework to address the above challenges based on the self-attention mechanism (SAM), referred to as UP-SAM. The contribution of this paper is three-fold. First, the self-attention mechanism is proposed where the spatial varying detail extraction and injection functions are estimated according to the attention representations indicating spectral characteristics of the MSI with sub-pixel accuracy. Second, such attention representations are derived from mixed pixels with the proposed stacked attention network powered with a stick-breaking structure to meet the physical constraints of mixed pixel formulations. Third, the detail extraction and injection functions are spatial varying based on the attention representations, which largely improves the reconstruction accuracy. Extensive experimental results demonstrate that the proposed approach is able to reconstruct sharper MSI of different types, with more details and less spectral distortion as compared to the state-of-the-art.

READ FULL TEXT

page 8

page 9

page 10

page 11

page 12

page 13

page 14

page 15

research
04/22/2023

SAWU-Net: Spatial Attention Weighted Unmixing Network for Hyperspectral Images

Hyperspectral unmixing is a critical yet challenging task in hyperspectr...
research
04/18/2023

Frequency Enhanced Hybrid Attention Network for Sequential Recommendation

The self-attention mechanism, which equips with a strong capability of m...
research
07/06/2021

Self-Adversarial Training incorporating Forgery Attention for Image Forgery Localization

Image editing techniques enable people to modify the content of an image...
research
07/25/2022

Deep dual stream residual network with contextual attention for pansharpening of remote sensing images

Pansharpening enhances spatial details of high spectral resolution multi...
research
03/04/2022

HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging

The rapid development of deep learning provides a better solution for th...
research
07/05/2023

Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks

This paper presents a deep learning-based spectral demosaicing technique...
research
07/01/2022

ChrSNet: Chromosome Straightening using Self-attention Guided Networks

Karyotyping is an important procedure to assess the possible existence o...

Please sign up or login with your details

Forgot password? Click here to reset