Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution

07/19/2022
by   Xiaoyu Dong, et al.
0

Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source. Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR.

READ FULL TEXT

page 2

page 10

page 14

page 22

page 25

page 26

page 27

page 28

research
03/02/2022

Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations

In this paper, we consider two challenging issues in reference-based sup...
research
11/09/2018

Cross and Learn: Cross-Modal Self-Supervision

In this paper we present a self-supervised method for representation lea...
research
04/04/2018

Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval

Thanks to the success of deep learning, cross-modal retrieval has made s...
research
06/15/2017

DOTE: Dual cOnvolutional filTer lEarning for Super-Resolution and Cross-Modality Synthesis in MRI

Cross-modal image synthesis is a topical problem in medical image comput...
research
01/19/2022

Self-Supervised Deep Blind Video Super-Resolution

Existing deep learning-based video super-resolution (SR) methods usually...
research
03/17/2023

Toward Super-Resolution for Appearance-Based Gaze Estimation

Gaze tracking is a valuable tool with a broad range of applications in v...
research
06/09/2020

Neural Methods for Point-wise Dependency Estimation

Since its inception, the neural estimation of mutual information (MI) ha...

Please sign up or login with your details

Forgot password? Click here to reset