Weakly Aligned Feature Fusion for Multimodal Object Detection

by   Lu Zhang, et al.

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method.


page 1

page 3

page 4

page 5

page 7

page 11

page 12

page 14


Multimodal Object Detection via Bayesian Fusion

Object detection with multimodal inputs can improve many safety-critical...

Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

Integrating multispectral data in object detection, especially visible a...

Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance

Image outpainting technology generates visually reasonable content regar...

Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments

Object detection is an essential task for autonomous robots operating in...

Cross-Modal Attentional Context Learning for RGB-D Object Detection

Recognizing objects from simultaneously sensed photometric (RGB) and dep...

Multimodal Learning for Hateful Memes Detection

Memes are multimedia documents containing images and phrases that usuall...

Reference-based Defect Detection Network

The defect detection task can be regarded as a realistic scenario of obj...

Please sign up or login with your details

Forgot password? Click here to reset