Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks

11/06/2021
by   Qinghui Liu, et al.
10

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.

READ FULL TEXT

page 2

page 12

page 13

page 16

page 17

page 19

research
06/02/2023

Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

In this paper, we introduce a novel Synchronized Class Token Fusion (SCT...
research
11/30/2021

Aerial Images Meet Crowdsourced Trajectories: A New Approach to Robust Road Extraction

Land remote sensing analysis is a crucial research in earth science. In ...
research
04/05/2023

Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping

Jointly harnessing complementary features of multi-modal input data in a...
research
06/01/2023

Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning

The development of federated learning (FL) methods, which aim to learn f...
research
08/10/2023

A Comparative Assessment of Multi-view fusion learning for Crop Classification

With a rapidly increasing amount and diversity of remote sensing (RS) da...
research
04/22/2023

Incomplete Multimodal Learning for Remote Sensing Data Fusion

The mechanism of connecting multimodal signals through self-attention op...
research
05/07/2020

Effective Data Fusion with Generalized Vegetation Index: Evidence from Land Cover Segmentation in Agriculture

How can we effectively leverage the domain knowledge from remote sensing...

Please sign up or login with your details

Forgot password? Click here to reset