DeepAI AI Chat
Log In Sign Up

Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution

by   Baoli Sun, et al.

Existing color-guided depth super-resolution (DSR) approaches require paired RGB-D data as training samples where the RGB image is used as structural guidance to recover the degraded depth map due to their geometrical similarity. However, the paired data may be limited or expensive to be collected in actual testing environment. Therefore, we explore for the first time to learn the cross-modality knowledge at training stage, where both RGB and depth modalities are available, but test on the target dataset, where only single depth modality exists. Our key idea is to distill the knowledge of scene structural guidance from RGB modality to the single DSR task without changing its network architecture. Specifically, we construct an auxiliary depth estimation (DE) task that takes an RGB image as input to estimate a depth map, and train both DSR task and DE task collaboratively to boost the performance of DSR. Upon this, a cross-task interaction module is proposed to realize bilateral cross task knowledge transfer. First, we design a cross-task distillation scheme that encourages DSR and DE networks to learn from each other in a teacher-student role-exchanging fashion. Then, we advance a structure prediction (SP) task that provides extra structure regularization to help both DSR and DE networks learn more informative structure representations for depth recovery. Extensive experiments demonstrate that our scheme achieves superior performance in comparison with other DSR methods.


page 2

page 3

page 7

page 9


360^∘ High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfer

Recently, omnidirectional images (ODIs) have become increasingly popular...

Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution

Color-guided depth super-resolution (DSR) is an encouraging paradigm tha...

PAG-Net: Progressive Attention Guided Depth Super-resolution Network

In this paper, we propose a novel method for the challenging problem of ...

Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge

Cross-modal knowledge distillation deals with transferring knowledge fro...

Teacher-Student Adversarial Depth Hallucination to Improve Face Recognition

We present the Teacher-Student Generative Adversarial Network (TS-GAN) t...

Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution

Guided depth map super-resolution (GDSR), as a hot topic in multi-modal ...

Translate-to-Recognize Networks for RGB-D Scene Recognition

Cross-modal transfer is helpful to enhance modality-specific discriminat...