1 Introduction
Correlative microscopies allow to combine different scales of observations and different contents, functional and morphological, based on the large panel of microscopy technologies available for life sciences [16] by registering datasets from different modalities together. The difference in scale of orders of magnitude, the level of accuracy required in biological questions, and the potential need of registration during the acquisition process call for automatic, robust and fast registration algorithm. Several approaches based on intensity matching have been proposed in the litterature based on specific metrics [14]
, potentially constructed using deep learning
[18, 13, 4], or using image analogies [7, 2]. One potential issue with intensitybased approaches is introducing bias by using the region of interest itself in the process of registration, and then assuming correlation in this region of interest. In addition, iterative approaches based on image data volumes of several gigabytes may be too heavy in computational and memory cost. For these reasons, we want to develop a generalized approach in 2 steps: targeted extraction of elements that are known by the biologist to be visible in both modalities; automatic matching of these elements to compute the transformation linking the targeted elements, that can then be applied to the dataset in ways less costly in memory than transforming the big image volumes. We used a point cloud based approach, where points from shapes or fiducials are extracted according to the biological questions and assumptions. In this work we focus on a point cloud registration method based on a paper by Huang et al. [8], and using algorithms from Papon et al. [9] and Wohlkinger and Vincze [17]. We compare our method to the classically used algorithm for point cloud registration ICP/Ransac [5, 1, 3].2 Proposed registration algorithm
2.1 Outline
The algorithm proceeds as follow: (1) We use a triangulation method to distribute points over surfaces that are detected in the 3d images. Then, (26) we follow a workflow proposed by Huang et al. [8] to register two point clouds: (2) We build a graph for each point cloud. The graph vertices and edges represent the global geometric structure of the point cloud; hopefully, this structure is robust to the possible irregularities or differences in the two point clouds, yet carries enough information to represent the biological objects. (3) We add features on the vertices and edges of the graph, to represent the local structure of the point clouds. (4) We solve a graph matching optimisation problem, which takes into account the structure of the graph as well as the features. (5) After the graph matching, we remove outliers, i.e., vertices that could not be satisfyingly matched. (6) Finally, we use the resulting mapping to find a geometric transform between the two sets of vertices, and hence between the two point clouds and the two images. For steps 2, 3 and 5, we rely on the PCL library [11] for the implementation.
2.2 Step by step
Generating point clouds.
In our setup, point clouds are generated either from the localization of specific landmarks in the image datasets (such as spot detections for beads used as landmarks) or from the Delaunay triangulation of an adhoc segmentation of common elements in the images (such as nuclei, cell or organs surface,…) using the methods described in PaulGilloteaux et al. [10]. In this paper we focus on the latter (Fig. 1).
Clustering on a point cloud and building a graph.
Following the recommendation of Huang et al. [8], we use supervoxel clustering [9] to build a graph from a point cloud. This algorithm is implemented by J. Papon as part of the PCL library [11]
. It consists in growing clusters of points from a set of seeds. The local clustering somewhat resembles the kmeans algorithm. The centres of the clusters will become the vertices of the graph; two vertices are linked by an edge if points in the corresponding clusters are sufficiently close. An important parameter of this algorithm is the seeding resolution, which directly impacts the lengths of the edges in the resulting graph. This parameter was selected based on the sampling resolution used to create the two point clouds.
To represent accurately the structure of the whole image, the graph must be connected. We add one vertex per connected component of the graph generated by Papon’s algorithm, and connect all these extra vertices.
Adding features on the vertices and edges of the graph.
During the clustering phase at the global level, a lot of information about the local structure of the point cloud was lost. Did the points in a cluster form a surface, or a volume? If it was a surface, what was its curvature? We summarize this information in the form of features on the graph vertices. Likewise, there should be features on the edges to recall the geometric structure of the graph in 3d space. For the features on the vertices, we use the ”Ensemble of Shape Functions” (ESF) descriptors, originally designed to be applied to a whole point cloud to match it against a database of known shapes, in the context of 3d image classification [17]. Here, we calculate the ESF descriptor of every cluster of points, and add it as a vertex feature on the graph. We use the implementation of ESF written by Wohlkinger for the PCL library [11].
Graph matching.
The problem of graph matching between two graphs and consists in finding a permutation that maps the vertices of to the vertices of in the most satisfying way. There are three criteria for satisfaction: (1) if is an edge in , then should be an edge in , and reciprocally; (2) the feature of a vertex in should be as close as possible to the feature of vertex in ; (3) the feature of an edge in should be as close as possible to the feature of edge in .
A permutation can be represented by a matrix of dimensions , where and are the numbers of vertices of and , respectively, and if and only if . The problem can then be expressed as a quadratic integer programming problem, i.e., an optimisation problem with linear constraints and quadratic objective function:
maximize:  (1)  
under constraints:  
(2)  
(3)  
(4) 
The objective function depends on the matrix defined by:

vertex feature similarity between and ;

edge feature similarity between and if they are edges in and respectively;

otherwise.
Thus the objective can be decomposed into two sums:
where the first sum accounts for the satisfaction of the matched vertices, and the second sum accounts for the satisfaction of the matched edges. Note that if , then ; so that the first sum could be made linear if necessary. Efficient algorithms to solve continuous quadratic problems are wellknown [6].
We relaxed the integer constraint into a linear constraint , and we tried solving this optimisation problem with two different solvers: (1) our own implementation of the FrankWolfe algorithm for quadratic programming [6]; (2) the nonlinear solver Ipopt [15]. FrankWolfe’s algorithm is based on the simplex algorithm and takes full advantage of the quadratic nature of the objective and the linear nature of the constraints. While FrankWolfe’s algorithm is welldocumented in the litterature, we could not find any satisfying implementation and had to reimplement it ourselves. In practice, we found that our implementation gets too often stuck in local optima. Ipopt’s solver, on the other hand, is a general solver for nonlinear problems, and cannot exploit the quadratic and linear natures of the objective and constraints. Its execution time is much longer than FrankWolfe’s algorithm, but it more often finds the optimal solution. With both solvers, we had the surprise to only ever find integer solutions. Although it might not be surprising that one of the optimal solutions is integral in some cases, litterature about graphmatching consistently argues that the difficulty of the problem resides in finding an integer solution [8, 19, 20]. We will investigate with more test cases the behaviour of these solvers.
Removing outliers.
During the graphmatching phase, some vertices were satisfyingly matched; it is likely that others couldn’t, because they have no equivalent in the graph of the other image. In addition, because the graphmatching step consists in maximising a sum of positive terms, if there are such vertices in both graphs, then these vertices will be matched together. Before using the matching to compute the transformation, it is important to remove these artificially matched vertices. For this, we resort to the Random Sample Consensus algorithm [5] and its implementation in the PCL library [11]. The idea of this algorithm is to find the largest set of vertices in one graph that can be mapped to their respective images in the other graph by the same rigid transform.
From matched nodes to geometric transform.
3 Preliminary results
We compare our algorithm with ICP/Ransac as implemented in the PCL library. Along with a rigid transform, we simulate difference in densities and missing parts of the point cloud and apply different known transforms. We register a 3d point cloud extracted from a segmented physiological structure, a mitral valve from a rat’s heart (Fig. 1), with transformed versions of this cloud (Fig. 2
). Note that in microscopy, the voxel size is usually known such that the cloud point coordinates can be expressed in micrometers in both source and target clouds and the relation between the point cloud can be constrained to be rigid. For each pair source cloud  transformed cloud, we estimate a transformation
using the algorithms, then compute the average distance where is the ground truth transformation and the number of points in the source. Results presented in Fig. 2 show that we outperfomed ICP/Ransac in all cases on this example.4 Conclusion
We implemented a method of registration for multimodal datasets based on point cloud registration. We have demonstrated on synthetic data obtained with simple transformations that the proposed method outperformed ICP/Ransac alone. Further simulation to represent the different cases encountered in correlated microscopies will be generated and analyzed, as well as validation on experimental data. Source code is available^{1}^{1}1https://github.com/skunne/pointcloudregistration/.
Acknowledgements
We acknowledge ANR18CE450015.
References
 [1] (1992) A method for registration of 3d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence. External Links: Link, Document Cited by: §1.
 [2] (2014) Multimodal registration for correlative microscopy using image analogies. Medical Image Analysis. External Links: Link, Document Cited by: §1.
 [3] (1991) Object modeling by registration of multiple range images. IEEE Robotics and Automation, Sacramento, USA. External Links: Link, Document Cited by: §1.
 [4] (2018) Deep similarity learning for multimodal medical images. CMBBE: Imaging & Visualization. External Links: Link, Document Cited by: §1.
 [5] (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM. Cited by: §1, §2.2.
 [6] (1956) An algorithm for quadratic programming. Naval research logistics quarterly. Cited by: §2.2, §2.2.
 [7] (2001) Image analogies. Comp. Graphics and Interactive Techniques, Los Angeles, USA. External Links: Link, Document Cited by: §1.
 [8] (2016) A systematic approach for crosssource point cloud registration by preserving macro and micro structures. IEEE Transactions on Image Processing. External Links: Document Cited by: §1, §2.1, §2.2, §2.2.

[9]
(2013)
Voxel cloud connectivity segmentation  supervoxels for point clouds.
IEEE Computer Vision and Pattern Recognition, Portland, USA
. External Links: Document Cited by: §1, §2.2.  [10] (2017) eCCLEM: flexible multidimensional registration software for correlative microscopies. Nature Methods. External Links: ISSN 15487091, Document Cited by: §2.2, §2.2.
 [11] (2011) 3D is here: point cloud library (PCL). IEEE Robotics and Automation (ICRA), Shanghai, China. External Links: Link, Document Cited by: §2.1, §2.2, §2.2, §2.2.
 [12] (1966) A generalized solution of the orthogonal procrustes problem. Psychometrika. External Links: ISSN 18600980, Document, Link Cited by: §2.2.
 [13] (2016) A deep metric for multimodal registration. Medical Image Comp. and Comp.Assisted Intervention, Athens, Greece. External Links: Link, Document Cited by: §1.
 [14] (2018) Intensitybased matching and registration for 3d correlative microscopy with large discrepancies. IEEE Biomedical Imaging. Cited by: §1.
 [15] (2006) On the implementation of an interiorpoint filter linesearch algorithm for largescale nonlinear programming. Math. Program.. External Links: Link, Document Cited by: §2.2.
 [16] (2020) Correlated multimodal imaging in life sciences: expanding the biomedical horizon.. Review, Front. Phys.  Medical Physics and Imaging. External Links: Document Cited by: §1.
 [17] (2011) Ensemble of shape functions for 3d object classification. IEEE Robotics and Biomimetics, Karon Beach, Thailand. External Links: Link, Document Cited by: §1, §2.2.

[18]
(2013)
Unsupervised deep feature learning for deformable registration of MR brain images
. Medical Image Comp. and Comp.Assisted Intervention, Nagoya, Japan. External Links: Link, Document Cited by: §1.  [19] (2008) A path following algorithm for the graph matching problem. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §2.2.
 [20] (2013) Deformable graph matching. IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA. Cited by: §2.2.
Comments
There are no comments yet.