2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds

by   Minhao Li, et al.

The commonly adopted detect-then-match approach to registration finds difficulties in the cross-modality cases due to the incompatible keypoint detection and inconsistent feature description. We propose, 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud and then extends them to form dense correspondences between pixels and points within the patch region. The coarse-level patch matching is based on transformer which jointly learns global contextual constraints with self-attention and cross-modality correlations with cross-attention. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level. Extensive experiments on two public benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art P2-Net by around 20 percentage points on inlier ratio and over 10 points on registration recall. Our code and models are available at https://github.com/minhaolee/2D3DMATR.


page 1

page 3

page 5

page 8

page 10

page 13

page 14


Geometric Transformer for Fast and Robust Point Cloud Registration

We study the problem of extracting accurate correspondences for point cl...

CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration

We study the problem of extracting correspondences between a pair of poi...

EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization

Visual localization is the task of estimating a 6-DoF camera pose of a q...

HybridPoint: Point Cloud Registration Based on Hybrid Point Sampling and Matching

Patch-to-point matching has become a robust way of point cloud registrat...

SGFeat: Salient Geometric Feature for Point Cloud Registration

Point Cloud Registration (PCR) is a critical and challenging task in com...

DEM Registration and Error Analysis using ASCII values

Digital Elevation Model (DEM), while providing a bare earth look, is hea...

TopicFM: Robust and Interpretable Feature Matching with Topic-assisted

Finding correspondences across images is an important task in many visua...

Please sign up or login with your details

Forgot password? Click here to reset