Cross-modal registration using point clouds and graph-matching in the context of correlative microscopies

by   Stephan Kunne, et al.

Correlative microscopy aims at combining two or more modalities to gain more information than the one provided by one modality on the same biological structure. Registration is needed at different steps of correlative microscopies workflows. Biologists want to select the image content used for registration not to introduce bias in the correlation of unknown structures. Intensity-based methods might not allow this selection and might be too slow when the images are very large. We propose an approach based on point clouds created from selected content by the biologist. These point clouds may be prone to big differences in densities but also missing parts and outliers. In this paper we present a method of registration for point clouds based on graph building and graph matching, and compare the method to iterative closest point based methods.



There are no comments yet.


page 1


A Graph-Matching Approach for Cross-view Registration of Over-view 2 and Street-view based Point Clouds

In this paper, based on the assumption that the object boundaries (e.g.,...

Augmented Semantic Signatures of Airborne LiDAR Point Clouds for Comparison

LiDAR point clouds provide rich geometric information, which is particul...

A Termination Criterion for Probabilistic PointClouds Registration

Probabilistic Point Clouds Registration (PPCR) is an algorithm that, in ...

Part2Word: Learning Joint Embedding of Point Clouds and Text by Matching Parts to Words

It is important to learn joint embedding for 3D shapes and text in diffe...

A Benchmark for Point Clouds Registration Algorithms

Point clouds registration is a fundamental step of many point clouds pro...

Computer-assisted polyp matching between optical colonoscopy and CT colonography: a phantom study

Potentially precancerous polyps detected with CT colonography (CTC) need...

Crowd Source Scene Change Detection and Local Map Update

As scene changes with time map descriptors become outdated, affecting VP...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Correlative microscopies allow to combine different scales of observations and different contents, functional and morphological, based on the large panel of microscopy technologies available for life sciences [16] by registering datasets from different modalities together. The difference in scale of orders of magnitude, the level of accuracy required in biological questions, and the potential need of registration during the acquisition process call for automatic, robust and fast registration algorithm. Several approaches based on intensity matching have been proposed in the litterature based on specific metrics [14]

, potentially constructed using deep learning 

[18, 13, 4], or using image analogies [7, 2]. One potential issue with intensity-based approaches is introducing bias by using the region of interest itself in the process of registration, and then assuming correlation in this region of interest. In addition, iterative approaches based on image data volumes of several gigabytes may be too heavy in computational and memory cost. For these reasons, we want to develop a generalized approach in 2 steps: targeted extraction of elements that are known by the biologist to be visible in both modalities; automatic matching of these elements to compute the transformation linking the targeted elements, that can then be applied to the dataset in ways less costly in memory than transforming the big image volumes. We used a point cloud based approach, where points from shapes or fiducials are extracted according to the biological questions and assumptions. In this work we focus on a point cloud registration method based on a paper by Huang et al. [8], and using algorithms from Papon et al. [9] and Wohlkinger and Vincze [17]. We compare our method to the classically used algorithm for point cloud registration ICP/Ransac [5, 1, 3].

2 Proposed registration algorithm

2.1 Outline

The algorithm proceeds as follow: (1) We use a triangulation method to distribute points over surfaces that are detected in the 3d images. Then, (2-6) we follow a workflow proposed by Huang et al. [8] to register two point clouds: (2) We build a graph for each point cloud. The graph vertices and edges represent the global geometric structure of the point cloud; hopefully, this structure is robust to the possible irregularities or differences in the two point clouds, yet carries enough information to represent the biological objects. (3) We add features on the vertices and edges of the graph, to represent the local structure of the point clouds. (4) We solve a graph matching optimisation problem, which takes into account the structure of the graph as well as the features. (5) After the graph matching, we remove outliers, i.e., vertices that could not be satisfyingly matched. (6) Finally, we use the resulting mapping to find a geometric transform between the two sets of vertices, and hence between the two point clouds and the two images. For steps 2, 3 and 5, we rely on the PCL library [11] for the implementation.

2.2 Step by step

Generating point clouds.

In our setup, point clouds are generated either from the localization of specific landmarks in the image datasets (such as spot detections for beads used as landmarks) or from the Delaunay triangulation of an ad-hoc segmentation of common elements in the images (such as nuclei, cell or organs surface,…) using the methods described in Paul-Gilloteaux et al. [10]. In this paper we focus on the latter (Fig. 1).

(a) (b) (c)
Figure 1: Example of data and generated graphs. (a) segmented valve extracted from microCT with points generated on its surface. (b) graph generated from this point cloud. (c) graph generated on a resampled and rotated version of the same point cloud.

Clustering on a point cloud and building a graph.

Following the recommendation of Huang et al. [8], we use supervoxel clustering [9] to build a graph from a point cloud. This algorithm is implemented by J. Papon as part of the PCL library [11]

. It consists in growing clusters of points from a set of seeds. The local clustering somewhat resembles the k-means algorithm. The centres of the clusters will become the vertices of the graph; two vertices are linked by an edge if points in the corresponding clusters are sufficiently close. An important parameter of this algorithm is the seeding resolution, which directly impacts the lengths of the edges in the resulting graph. This parameter was selected based on the sampling resolution used to create the two point clouds.

To represent accurately the structure of the whole image, the graph must be connected. We add one vertex per connected component of the graph generated by Papon’s algorithm, and connect all these extra vertices.

Adding features on the vertices and edges of the graph.

During the clustering phase at the global level, a lot of information about the local structure of the point cloud was lost. Did the points in a cluster form a surface, or a volume? If it was a surface, what was its curvature? We summarize this information in the form of features on the graph vertices. Likewise, there should be features on the edges to recall the geometric structure of the graph in 3d space. For the features on the vertices, we use the ”Ensemble of Shape Functions” (ESF) descriptors, originally designed to be applied to a whole point cloud to match it against a database of known shapes, in the context of 3d image classification [17]. Here, we calculate the ESF descriptor of every cluster of points, and add it as a vertex feature on the graph. We use the implementation of ESF written by Wohlkinger for the PCL library [11].

Graph matching.

The problem of graph matching between two graphs and consists in finding a permutation that maps the vertices of to the vertices of in the most satisfying way. There are three criteria for satisfaction: (1) if is an edge in , then should be an edge in , and reciprocally; (2) the feature of a vertex in should be as close as possible to the feature of vertex in ; (3) the feature of an edge in should be as close as possible to the feature of edge in .

A permutation can be represented by a matrix of dimensions , where and are the numbers of vertices of and , respectively, and if and only if . The problem can then be expressed as a quadratic integer programming problem, i.e., an optimisation problem with linear constraints and quadratic objective function:

maximize: (1)
under constraints:

The objective function depends on the matrix defined by:

  • vertex feature similarity between and ;

  • edge feature similarity between and if they are edges in and respectively;

  • otherwise.

Thus the objective can be decomposed into two sums:

where the first sum accounts for the satisfaction of the matched vertices, and the second sum accounts for the satisfaction of the matched edges. Note that if , then ; so that the first sum could be made linear if necessary. Efficient algorithms to solve continuous quadratic problems are well-known [6].

We relaxed the integer constraint into a linear constraint , and we tried solving this optimisation problem with two different solvers: (1) our own implementation of the Frank-Wolfe algorithm for quadratic programming [6]; (2) the nonlinear solver Ipopt [15]. Frank-Wolfe’s algorithm is based on the simplex algorithm and takes full advantage of the quadratic nature of the objective and the linear nature of the constraints. While Frank-Wolfe’s algorithm is well-documented in the litterature, we could not find any satisfying implementation and had to reimplement it ourselves. In practice, we found that our implementation gets too often stuck in local optima. Ipopt’s solver, on the other hand, is a general solver for nonlinear problems, and cannot exploit the quadratic and linear natures of the objective and constraints. Its execution time is much longer than Frank-Wolfe’s algorithm, but it more often finds the optimal solution. With both solvers, we had the surprise to only ever find integer solutions. Although it might not be surprising that one of the optimal solutions is integral in some cases, litterature about graph-matching consistently argues that the difficulty of the problem resides in finding an integer solution [8, 19, 20]. We will investigate with more test cases the behaviour of these solvers.

Removing outliers.

During the graph-matching phase, some vertices were satisfyingly matched; it is likely that others couldn’t, because they have no equivalent in the graph of the other image. In addition, because the graph-matching step consists in maximising a sum of positive terms, if there are such vertices in both graphs, then these vertices will be matched together. Before using the matching to compute the transformation, it is important to remove these artificially matched vertices. For this, we resort to the Random Sample Consensus algorithm [5] and its implementation in the PCL library [11]. The idea of this algorithm is to find the largest set of vertices in one graph that can be mapped to their respective images in the other graph by the same rigid transform.

From matched nodes to geometric transform.

From the matched vertex sets, we compute a rigid transformation using the Schönemann method [12] as implemented in eC-CLEM [10].

3 Preliminary results

Figure 2: Mean distance between points in micrometers in registered source point cloud and destination point cloud. The source point cloud is always the same, an artificial cloud mimicking cardiac valve. The ground truth transforms are: identity; rotation in plane . S denotes that the destination cloud is resampled by discarding of the points. C denotes that half of the destination cloud is cropped out.

We compare our algorithm with ICP/Ransac as implemented in the PCL library. Along with a rigid transform, we simulate difference in densities and missing parts of the point cloud and apply different known transforms. We register a 3d point cloud extracted from a segmented physiological structure, a mitral valve from a rat’s heart (Fig. 1), with transformed versions of this cloud (Fig. 2

). Note that in microscopy, the voxel size is usually known such that the cloud point coordinates can be expressed in micrometers in both source and target clouds and the relation between the point cloud can be constrained to be rigid. For each pair source cloud - transformed cloud, we estimate a transformation

using the algorithms, then compute the average distance where is the ground truth transformation and the number of points in the source. Results presented in Fig. 2 show that we outperfomed ICP/Ransac in all cases on this example.

4 Conclusion

We implemented a method of registration for multimodal datasets based on point cloud registration. We have demonstrated on synthetic data obtained with simple transformations that the proposed method outperformed ICP/Ransac alone. Further simulation to represent the different cases encountered in correlated microscopies will be generated and analyzed, as well as validation on experimental data. Source code is available111


We acknowledge ANR-18-CE45-0015.


  • [1] P. J. Besl and N. D. McKay (1992) A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence. External Links: Link, Document Cited by: §1.
  • [2] T. Cao, C. Zach, S. Modla, D. Powell, K. Czymmek, and M. Niethammer (2014) Multi-modal registration for correlative microscopy using image analogies. Medical Image Analysis. External Links: Link, Document Cited by: §1.
  • [3] Y. Chen and G. G. Medioni (1991) Object modeling by registration of multiple range images. IEEE Robotics and Automation, Sacramento, USA. External Links: Link, Document Cited by: §1.
  • [4] X. Cheng, L. Zhang, and Y. Zheng (2018) Deep similarity learning for multimodal medical images. CMBBE: Imaging & Visualization. External Links: Link, Document Cited by: §1.
  • [5] M. A. Fischler and R. C. Bolles (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM. Cited by: §1, §2.2.
  • [6] M. Frank and P. Wolfe (1956) An algorithm for quadratic programming. Naval research logistics quarterly. Cited by: §2.2, §2.2.
  • [7] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. Salesin (2001) Image analogies. Comp. Graphics and Interactive Techniques, Los Angeles, USA. External Links: Link, Document Cited by: §1.
  • [8] X. Huang, J. Zhang, L. Fan, Q. Wu, and C. Yuan (2016) A systematic approach for cross-source point cloud registration by preserving macro and micro structures. IEEE Transactions on Image Processing. External Links: Document Cited by: §1, §2.1, §2.2, §2.2.
  • [9] J. Papon, A. Abramov, M. Schoeler, and F. Wörgötter (2013) Voxel cloud connectivity segmentation - supervoxels for point clouds.

    IEEE Computer Vision and Pattern Recognition, Portland, USA

    External Links: Document Cited by: §1, §2.2.
  • [10] P. Paul-Gilloteaux, X. Heiligenstein, M. Belle, M. Domart, B. Larijani, L. Collinson, G. Raposo, and J. Salamero (2017) eC-CLEM: flexible multidimensional registration software for correlative microscopies. Nature Methods. External Links: ISSN 1548-7091, Document Cited by: §2.2, §2.2.
  • [11] R. B. Rusu and S. Cousins (2011) 3D is here: point cloud library (PCL). IEEE Robotics and Automation (ICRA), Shanghai, China. External Links: Link, Document Cited by: §2.1, §2.2, §2.2, §2.2.
  • [12] P. H. Schönemann (1966) A generalized solution of the orthogonal procrustes problem. Psychometrika. External Links: ISSN 1860-0980, Document, Link Cited by: §2.2.
  • [13] M. Simonovsky, B. Gutiérrez-Becker, D. Mateus, N. Navab, and N. Komodakis (2016) A deep metric for multimodal registration. Medical Image Comp. and Comp.-Assisted Intervention, Athens, Greece. External Links: Link, Document Cited by: §1.
  • [14] B. M. Toledo Acosta, X. Heiligenstein, G. Malandain, and P. Bouthemy (2018) Intensity-based matching and registration for 3d correlative microscopy with large discrepancies. IEEE Biomedical Imaging. Cited by: §1.
  • [15] A. Wächter and L. T. Biegler (2006) On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program.. External Links: Link, Document Cited by: §2.2.
  • [16] A. Walter, P. Paul-Gilloteaux, B. Plochberger, L. Sefc, P. Verkade, J. Mannheim, P. Slezak, A. Unterhuber, M. Marchetti-Deschmann, M. Ogri, K. Bühler, D. Fixler, S. Geyer, W. Weninge, M. Glösmann, S. Handschuh, and T. Wanek (2020) Correlated multimodal imaging in life sciences: expanding the biomedical horizon.. Review, Front. Phys. - Medical Physics and Imaging. External Links: Document Cited by: §1.
  • [17] W. Wohlkinger and M. Vincze (2011) Ensemble of shape functions for 3d object classification. IEEE Robotics and Biomimetics, Karon Beach, Thailand. External Links: Link, Document Cited by: §1, §2.2.
  • [18] G. Wu, M. Kim, Q. Wang, Y. Gao, S. Liao, and D. Shen (2013)

    Unsupervised deep feature learning for deformable registration of MR brain images

    Medical Image Comp. and Comp.-Assisted Intervention, Nagoya, Japan. External Links: Link, Document Cited by: §1.
  • [19] M. Zaslavskiy, F. Bach, and J. Vert (2008) A path following algorithm for the graph matching problem. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2.2.
  • [20] F. Zhou and F. De la Torre (2013) Deformable graph matching. IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA. Cited by: §2.2.