Correlative microscopies allow to combine different scales of observations and different contents, functional and morphological, based on the large panel of microscopy technologies available for life sciences  by registering datasets from different modalities together. The difference in scale of orders of magnitude, the level of accuracy required in biological questions, and the potential need of registration during the acquisition process call for automatic, robust and fast registration algorithm. Several approaches based on intensity matching have been proposed in the litterature based on specific metrics 
, potentially constructed using deep learning[18, 13, 4], or using image analogies [7, 2]. One potential issue with intensity-based approaches is introducing bias by using the region of interest itself in the process of registration, and then assuming correlation in this region of interest. In addition, iterative approaches based on image data volumes of several gigabytes may be too heavy in computational and memory cost. For these reasons, we want to develop a generalized approach in 2 steps: targeted extraction of elements that are known by the biologist to be visible in both modalities; automatic matching of these elements to compute the transformation linking the targeted elements, that can then be applied to the dataset in ways less costly in memory than transforming the big image volumes. We used a point cloud based approach, where points from shapes or fiducials are extracted according to the biological questions and assumptions. In this work we focus on a point cloud registration method based on a paper by Huang et al. , and using algorithms from Papon et al.  and Wohlkinger and Vincze . We compare our method to the classically used algorithm for point cloud registration ICP/Ransac [5, 1, 3].
2 Proposed registration algorithm
The algorithm proceeds as follow: (1) We use a triangulation method to distribute points over surfaces that are detected in the 3d images. Then, (2-6) we follow a workflow proposed by Huang et al.  to register two point clouds: (2) We build a graph for each point cloud. The graph vertices and edges represent the global geometric structure of the point cloud; hopefully, this structure is robust to the possible irregularities or differences in the two point clouds, yet carries enough information to represent the biological objects. (3) We add features on the vertices and edges of the graph, to represent the local structure of the point clouds. (4) We solve a graph matching optimisation problem, which takes into account the structure of the graph as well as the features. (5) After the graph matching, we remove outliers, i.e., vertices that could not be satisfyingly matched. (6) Finally, we use the resulting mapping to find a geometric transform between the two sets of vertices, and hence between the two point clouds and the two images. For steps 2, 3 and 5, we rely on the PCL library  for the implementation.
2.2 Step by step
Generating point clouds.
In our setup, point clouds are generated either from the localization of specific landmarks in the image datasets (such as spot detections for beads used as landmarks) or from the Delaunay triangulation of an ad-hoc segmentation of common elements in the images (such as nuclei, cell or organs surface,…) using the methods described in Paul-Gilloteaux et al. . In this paper we focus on the latter (Fig. 1).
Clustering on a point cloud and building a graph.
. It consists in growing clusters of points from a set of seeds. The local clustering somewhat resembles the k-means algorithm. The centres of the clusters will become the vertices of the graph; two vertices are linked by an edge if points in the corresponding clusters are sufficiently close. An important parameter of this algorithm is the seeding resolution, which directly impacts the lengths of the edges in the resulting graph. This parameter was selected based on the sampling resolution used to create the two point clouds.
To represent accurately the structure of the whole image, the graph must be connected. We add one vertex per connected component of the graph generated by Papon’s algorithm, and connect all these extra vertices.
Adding features on the vertices and edges of the graph.
During the clustering phase at the global level, a lot of information about the local structure of the point cloud was lost. Did the points in a cluster form a surface, or a volume? If it was a surface, what was its curvature? We summarize this information in the form of features on the graph vertices. Likewise, there should be features on the edges to recall the geometric structure of the graph in 3d space. For the features on the vertices, we use the ”Ensemble of Shape Functions” (ESF) descriptors, originally designed to be applied to a whole point cloud to match it against a database of known shapes, in the context of 3d image classification . Here, we calculate the ESF descriptor of every cluster of points, and add it as a vertex feature on the graph. We use the implementation of ESF written by Wohlkinger for the PCL library .
The problem of graph matching between two graphs and consists in finding a permutation that maps the vertices of to the vertices of in the most satisfying way. There are three criteria for satisfaction: (1) if is an edge in , then should be an edge in , and reciprocally; (2) the feature of a vertex in should be as close as possible to the feature of vertex in ; (3) the feature of an edge in should be as close as possible to the feature of edge in .
A permutation can be represented by a matrix of dimensions , where and are the numbers of vertices of and , respectively, and if and only if . The problem can then be expressed as a quadratic integer programming problem, i.e., an optimisation problem with linear constraints and quadratic objective function:
The objective function depends on the matrix defined by:
vertex feature similarity between and ;
edge feature similarity between and if they are edges in and respectively;
Thus the objective can be decomposed into two sums:
where the first sum accounts for the satisfaction of the matched vertices, and the second sum accounts for the satisfaction of the matched edges. Note that if , then ; so that the first sum could be made linear if necessary. Efficient algorithms to solve continuous quadratic problems are well-known .
We relaxed the integer constraint into a linear constraint , and we tried solving this optimisation problem with two different solvers: (1) our own implementation of the Frank-Wolfe algorithm for quadratic programming ; (2) the nonlinear solver Ipopt . Frank-Wolfe’s algorithm is based on the simplex algorithm and takes full advantage of the quadratic nature of the objective and the linear nature of the constraints. While Frank-Wolfe’s algorithm is well-documented in the litterature, we could not find any satisfying implementation and had to reimplement it ourselves. In practice, we found that our implementation gets too often stuck in local optima. Ipopt’s solver, on the other hand, is a general solver for nonlinear problems, and cannot exploit the quadratic and linear natures of the objective and constraints. Its execution time is much longer than Frank-Wolfe’s algorithm, but it more often finds the optimal solution. With both solvers, we had the surprise to only ever find integer solutions. Although it might not be surprising that one of the optimal solutions is integral in some cases, litterature about graph-matching consistently argues that the difficulty of the problem resides in finding an integer solution [8, 19, 20]. We will investigate with more test cases the behaviour of these solvers.
During the graph-matching phase, some vertices were satisfyingly matched; it is likely that others couldn’t, because they have no equivalent in the graph of the other image. In addition, because the graph-matching step consists in maximising a sum of positive terms, if there are such vertices in both graphs, then these vertices will be matched together. Before using the matching to compute the transformation, it is important to remove these artificially matched vertices. For this, we resort to the Random Sample Consensus algorithm  and its implementation in the PCL library . The idea of this algorithm is to find the largest set of vertices in one graph that can be mapped to their respective images in the other graph by the same rigid transform.
From matched nodes to geometric transform.
3 Preliminary results
We compare our algorithm with ICP/Ransac as implemented in the PCL library. Along with a rigid transform, we simulate difference in densities and missing parts of the point cloud and apply different known transforms. We register a 3d point cloud extracted from a segmented physiological structure, a mitral valve from a rat’s heart (Fig. 1), with transformed versions of this cloud (Fig. 2
). Note that in microscopy, the voxel size is usually known such that the cloud point coordinates can be expressed in micrometers in both source and target clouds and the relation between the point cloud can be constrained to be rigid. For each pair source cloud - transformed cloud, we estimate a transformationusing the algorithms, then compute the average distance where is the ground truth transformation and the number of points in the source. Results presented in Fig. 2 show that we outperfomed ICP/Ransac in all cases on this example.
We implemented a method of registration for multimodal datasets based on point cloud registration. We have demonstrated on synthetic data obtained with simple transformations that the proposed method outperformed ICP/Ransac alone. Further simulation to represent the different cases encountered in correlated microscopies will be generated and analyzed, as well as validation on experimental data. Source code is available111https://github.com/skunne/pointcloudregistration/.
We acknowledge ANR-18-CE45-0015.
-  (1992) A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence. External Links: Cited by: §1.
-  (2014) Multi-modal registration for correlative microscopy using image analogies. Medical Image Analysis. External Links: Cited by: §1.
-  (1991) Object modeling by registration of multiple range images. IEEE Robotics and Automation, Sacramento, USA. External Links: Cited by: §1.
-  (2018) Deep similarity learning for multimodal medical images. CMBBE: Imaging & Visualization. External Links: Cited by: §1.
-  (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM. Cited by: §1, §2.2.
-  (1956) An algorithm for quadratic programming. Naval research logistics quarterly. Cited by: §2.2, §2.2.
-  (2001) Image analogies. Comp. Graphics and Interactive Techniques, Los Angeles, USA. External Links: Cited by: §1.
-  (2016) A systematic approach for cross-source point cloud registration by preserving macro and micro structures. IEEE Transactions on Image Processing. External Links: Cited by: §1, §2.1, §2.2, §2.2.
-  (2013) Voxel cloud connectivity segmentation - supervoxels for point clouds. . External Links: Cited by: §1, §2.2.
-  (2017) eC-CLEM: flexible multidimensional registration software for correlative microscopies. Nature Methods. External Links: Cited by: §2.2, §2.2.
-  (2011) 3D is here: point cloud library (PCL). IEEE Robotics and Automation (ICRA), Shanghai, China. External Links: Cited by: §2.1, §2.2, §2.2, §2.2.
-  (1966) A generalized solution of the orthogonal procrustes problem. Psychometrika. External Links: Cited by: §2.2.
-  (2016) A deep metric for multimodal registration. Medical Image Comp. and Comp.-Assisted Intervention, Athens, Greece. External Links: Cited by: §1.
-  (2018) Intensity-based matching and registration for 3d correlative microscopy with large discrepancies. IEEE Biomedical Imaging. Cited by: §1.
-  (2006) On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program.. External Links: Cited by: §2.2.
-  (2020) Correlated multimodal imaging in life sciences: expanding the biomedical horizon.. Review, Front. Phys. - Medical Physics and Imaging. External Links: Cited by: §1.
-  (2011) Ensemble of shape functions for 3d object classification. IEEE Robotics and Biomimetics, Karon Beach, Thailand. External Links: Cited by: §1, §2.2.
Unsupervised deep feature learning for deformable registration of MR brain images. Medical Image Comp. and Comp.-Assisted Intervention, Nagoya, Japan. External Links: Cited by: §1.
-  (2008) A path following algorithm for the graph matching problem. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2.2.
-  (2013) Deformable graph matching. IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA. Cited by: §2.2.