Boosting Image Super-Resolution Via Fusion of Complementary Information Captured by Multi-Modal Sensors

by   Fan Wang, et al.

Image Super-Resolution (SR) provides a promising technique to enhance the image quality of low-resolution optical sensors, facilitating better-performing target detection and autonomous navigation in a wide range of robotics applications. It is noted that the state-of-the-art SR methods are typically trained and tested using single-channel inputs, neglecting the fact that the cost of capturing high-resolution images in different spectral domains varies significantly. In this paper, we attempt to leverage complementary information from a low-cost channel (visible/depth) to boost image quality of an expensive channel (thermal) using fewer parameters. To this end, we first present an effective method to virtually generate pixel-wise aligned visible and thermal images based on real-time 3D reconstruction of multi-modal data captured at various viewpoints. Then, we design a feature-level multispectral fusion residual network model to perform high-accuracy SR of thermal images by adaptively integrating co-occurrence features presented in multispectral images. Experimental results demonstrate that this new approach can effectively alleviate the ill-posed inverse problem of image SR by taking into account complementary information from an additional low-cost channel, significantly outperforming state-of-the-art SR approaches in terms of both accuracy and efficiency.


page 1

page 2

page 3

page 4

page 5

page 6

page 7


Hybrid Pixel-Unshuffled Network for Lightweight Image Super-Resolution

Convolutional neural network (CNN) has achieved great success on image s...

Thermal Image Super-Resolution Using Second-Order Channel Attention with Varying Receptive Fields

Thermal images model the long-infrared range of the electromagnetic spec...

Scene Text Image Super-Resolution in the Wild

Low-resolution text images are often seen in natural scenes such as docu...

Fusion of Deep and Non-Deep Methods for Fast Super-Resolution of Satellite Images

In the emerging commercial space industry there is a drastic increase in...

Multi-FAN: Multi-Spectral Mosaic Super-Resolution Via Multi-Scale Feature Aggregation Network

This paper introduces a novel method to super-resolve multi-spectral ima...

Photothermal-SR-Net: A Customized Deep Unfolding Neural Network for Photothermal Super Resolution Imaging

This paper presents deep unfolding neural networks to handle inverse pro...

Multimodal Sensor Fusion In Single Thermal image Super-Resolution

With the fast growth in the visual surveillance and security sectors, th...