Adaptive Context-Aware Multi-Modal Network for Depth Completion

08/25/2020
by   Shanshan Zhao, et al.
0

Depth completion aims to recover a dense depth map from the sparse depth data and the corresponding single RGB image. The observed pixels provide the significant guidance for the recovery of the unobserved pixels' depth. However, due to the sparsity of the depth data, the standard convolution operation, exploited by most of existing methods, is not effective to model the observed contexts with depth values. To address this issue, we propose to adopt the graph propagation to capture the observed spatial contexts. Specifically, we first construct multiple graphs at different scales from observed pixels. Since the graph structure varies from sample to sample, we then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Furthermore, considering the mutli-modality of input data, we exploit the graph propagation on the two modalities respectively to extract multi-modal representations. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. The proposed strategy preserves the original information for one modality and also absorbs complementary information from the other through learning the adaptive gating weights. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks, i.e., KITTI and NYU-v2, and at the same time has fewer parameters than latest models. Our code is available at: <https://github.com/sshan-zhao/ACMNet>.

READ FULL TEXT

page 2

page 5

page 9

page 11

research
08/31/2022

NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

Multi-modal MR imaging is routinely used in clinical practice to diagnos...
research
04/27/2023

Adaptive-Mask Fusion Network for Segmentation of Drivable Road and Negative Obstacle With Untrustworthy Features

Segmentation of drivable roads and negative obstacles is critical to the...
research
07/21/2021

Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation

Liver cancer is one of the most common cancers worldwide. Due to inconsp...
research
11/13/2019

CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion

Depth Completion deals with the problem of converting a sparse depth map...
research
07/21/2022

MetaComp: Learning to Adapt for Online Depth Completion

Relying on deep supervised or self-supervised learning, previous methods...
research
09/05/2023

Decomposed Guided Dynamic Filters for Efficient RGB-Guided Depth Completion

RGB-guided depth completion aims at predicting dense depth maps from spa...
research
05/03/2021

Multi-modal Bifurcated Network for Depth Guided Image Relighting

Image relighting aims to recalibrate the illumination setting in an imag...

Please sign up or login with your details

Forgot password? Click here to reset