Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

11/15/2021
by   Yuanhao Cai, et al.
0

Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S-MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs.

READ FULL TEXT

page 3

page 4

page 6

page 7

page 8

research
03/09/2022

Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

Many algorithms have been developed to solve the inverse problem of code...
research
04/17/2022

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction

Existing leading methods for spectral reconstruction (SR) focus on desig...
research
09/24/2022

S^2-Transformer for Mask-Aware Hyperspectral Image Reconstruction

The technology of hyperspectral imaging (HSI) records the visual informa...
research
05/06/2023

SST-ReversibleNet: Reversible-prior-based Spectral-Spatial Transformer for Efficient Hyperspectral Image Reconstruction

Spectral image reconstruction is an important task in snapshot compresse...
research
12/31/2021

Calibrated Hyperspectral Image Reconstruction via Graph-based Self-Tuning Network

Recently, hyperspectral imaging (HSI) has attracted increasing research ...
research
01/06/2022

Flow-Guided Sparse Transformer for Video Deblurring

Exploiting similar and sharper scene patches in spatio-temporal neighbor...
research
08/03/2023

DMDC: Dynamic-mask-based dual camera design for snapshot Hyperspectral Imaging

Deep learning methods are developing rapidly in coded aperture snapshot ...

Please sign up or login with your details

Forgot password? Click here to reset