The alignment of two point sets is a fundamental geometric problem that occurs in many computer vision and robotics applications. In computer vision, the technique is used to stitch together partial 3D reconstructions in order to form a more complete model of an object or environment. In robotics, point set registration is an essential ingredient to simultaneous localization and mapping with affordable consumer depth cameras  or powerful 3D laser range scanners . The general approach for aligning two point sets does not require initial correspondences. It is given by the Iterative Closest Point (ICP) algorithm 
, a local search strategy that alternates between geometric correspondence establishment (i.e. by simple nearest neighbour search) and Procrustes alignment. The iterative procedure depends on a sufficiently accurate initial guess about the relative transformation (e.g. an identity transformation in the case of incremental ego-motion estimation).
The present work is motivated by a common problem that occurs when performing a dense reconstruction of an environment which contains multiple instances of the same object. An example of the latter is given by a room in which the same type of chair occurs more than once. Let us assume that the front-end of our reconstruction framework encompasses semantic recognition capabilities which are used to segment out partial point sets of objects of the same class and type . There is a general interest in aligning those partial object point sets towards exploiting their mutual information and completing or even improving the reconstruction of each instance. The difficulty of this partial point set registration problem arises from two factors:
The relative pose between the different objects is arbitrary and unknown.
Since the objects are observed under an arbitrary pose and with potential occlusions, the measured partial point sets potentially have very little overlap.
Our contribution focuses on the registration of only two partial point sets, which appears as a worthwhile starting point given that the registration of more than two point sets may be broken down into many pair-wise alignments. The plain ICP algorithm is only a local search strategy that depends on a sufficiently accurate initial guess, which is why it may not serve as a valid solution to our problem. A potential remedy is given by the globally optimal ICP algorithm presented by Yang et al. . However, the algorithm still depends on sufficient overlap in the partial point sets, which is not necessarily a given (50% is reported as a requirement for high success rate).
The core idea of the present work consists of exploiting the fact that the majority of commonly observed objects contain a plane of symmetry. By reflecting the points of each partial point set with respect to the plane of symmetry, we may effectively increase the overlap between the two sets and vastly improve the success rate of the registration process. However, given that each point set only observes part of the object, the identification of the plane of symmetry in each individual point set appears to be an equally difficult and ill-posed problem than the partial point set registration problem itself. It is only after the aligning transformation is found that symmetry plane detection would become a less challenging problem. In conclusion, the solution of each problem strongly depends on a prior solution to the other. We therefore present the following contributions:
We solve this chicken-and-egg problem by a joint solution of the aligning transformation as well as the symmetry plane parameters. To the best of our knowledge, our work is the first to address those two problems jointly, thus leading to a vast improvement over the existing state-of-the-art in globally optimal point set registration, especially in situations in which two point sets contain very limited overlap.
We immediately provide a globally optimal solution to this problem by employing the branch-and-bound optimisation paradigm. Our work implicitly provides the first solution to globally optimal symmetry plane estimation in a single point set, or—more generally—symmetry plane detection across two point sets.
We show a meaningful application of our algorithm in a dense 3D reconstruction scenario in which multiple instances of the same object occur.
2 Related Work
Despite strong mutual dependency, our method is the first to perform joint point set alignment and symmetry plane estimation. Our literature review therefore cites prior art on those two topics individually.
2.1 Point-set Registration
The Iterative Closest Point (ICP) [6, 3, 28] algorithm is a popular method for aligning two point sets. It does not depend on a prior derivation of point-to-point correspondences, and simply aligns the two sets by iteratively alternating between the two steps of finding nearest neighbours (e.g. by minimising point-to-point distances), and computing the alignment (e.g. using Arun’s method 
). To improve robustness of the algorithm against occlusions and reduced overlap, the method has been extended by outlier rejection[28, 12] or data trimming  techniques. However, the classical ICP algorithm remains a local search algorithm for which the convergence depends on initial guess and sufficient overlap between both point sets.
An entire family of alternative approaches relies on the idea of expressing both point sets by a Gaussian Mixture Model (GMM) and aligning the latter using Gaussian Mixture Alignment (GMA). Notable GMA-based techniques for rigid and non-rigid registration are given by the robust point matching algorithm by Chui and Rangarajan, the coherent point drift strategy by Myronenko and Song , kernel correlation by Tsin and Kanade , and the GMMReg algorithm by Jian and Vemuri . While GMA is advertised by improved robustness against poor initialisations, noise, and outliers, another key advantage with respect to point-based methods is given by a closed-form expression to evaluate the quality of the alignment (i.e. the method does not depend on an alternating search for nearest neighbours). However, the listed GMA algorithms remain local search algorithms, which makes them inapplicable in scenarios in which no prior about the relative transformation is given upfront.
In contrast, globally optimal algorithms avoid local minima by searching the entire space of relative transformations, often using the branch-and-bound paradigm [17, 21, 22]. Yang et al. [27, 26] propose the Go-ICP algorithm, which applies branch-and-bound to the ICP problem to find the globally optimal minimum of the sum of L2-distances between nearest neighbours from two aligned point sets. The method is accelerated by using local ICP in the loop. However, missing robustness of the cost-function causes the method to remain sensitive with respect to occlusions and partial overlaps. Campbell et al.  finally devise GOGMA, a branch-and-bound variant in which the objective of minimising point-to-point errors is again replaced by the convolution of GMMs.
Inspired by those recent advances, we also employ the nested branch-and-bound strategy integrated with local ICP to find a globally optimal alignment. However, in contrast to all prior art, we are the first to jointly fit a plane of symmetry, which leads to a large improvement in partial scan alignment for common symmetric objects.
2.2 Symmetry Plane Estimation
For a comprehensive review of symmetry detection, we kindly refer the reader to Liu et al.’s review . Here we only focus on the problem of symmetry plane fitting with missing data. The most straightforward solution is given by employing the RANdom SAmple Consensus (RANSAC) algorithm proposed by Fischler and Bolles , a well-known algorithm for robust model fitting for outlier affected data. In the context of shape matching, the basic idea is to extract sparse characteristic points and match them between both sets. We then choose a random subset of correspondences and derive a hypothesis for the global transformation induced by these samples. The alignment quality is finally evaluated by the matching error between the two shapes. The method can be easily applied for detecting a plane of symmetry in a single point set by hypothesising the latter to be orthogonal to the axis connecting a correspondence. For example, Cohen et al.  detect symmetries in sparse point clouds by using appearance-based 3D-3D point correspondences in a RANSAC scheme. The detected symmetries are subsequently explored to eliminate noise from the point-clouds. Xu et al.  present a voting algorithm to detect the intrinsic reflectional symmetry axis. Using the axis as a hint, a completion algorithm for missing geometry is again shown. Jiang et al.  on the other hand propose an algorithm to find intrinsic symmetries in point clouds by using a curve skeleton. A set of filters then produces a candidate set of symmetric correspondences which are finally verified via spectral analysis. Although these works show results on partial data, the amount of missing data is typically small. Inspired by the work of  which detects symmetry by registration, we propose to detect the symmetry plane alongside partial point set registration, thus leading to improved performance in situations with limited overlap.
We start by introducing the notation used throughout the paper and review the basic formulation of the ICP problem as well as planar reflections.
3.1 Notations and Assumptions
Let us denote the two partial object point sets by and (sometimes called the model and data point sets, respectively). The goal pursued in this paper is the identification of a Euclidean transformation given by the rotation and the translation that transforms the points of such that they align with the points of . If and contain points in the 2D plane, and form an element of the group . If and contain 3D points, and will be an element of . Note that alignment denotes a more general idea rather than just the minimisation of the sum of distances between each point of and its closest point within . The point sets have different cardinality and potentially observe very different parts of the object with only very little overlap. This motivates our approach that takes object symmetries into account.
3.2 Registration of Two Point Sets
The standard solution to the point set registration problem is given by the ICP algorithm , which minimizes the alignment error given by
where is the per-point residual error for , and is the closest point to within , i.e.
Given an initial transformation and , the ICP algorithm iteratively solves the above minimization problem by alternating between updating the aligning transformation with fixed (i.e. using (1)), and updating the closest-point matches themselves using (2). It is intuitively clear that the ICP algorithm only convergence to a local minimum.
3.3 Modelling and Identifying Symmetry
Symmetry is modelled by a reflection by the symmetry plane. Let us define the symmetry plane by the normal and depth-of-plane such that any point on the plane satisfies the relation . Let now be a (2D or 3D) point from . The reflection plane reflects to a single reflected point given by
The term in parentheses is the signed distance between and the reflection plane. The subtraction of times this distance reflects the point to the other side of the plane. The problem of symmetry identification may now be formulated as a minimisation of the symmetry distance defined by
where is the per-point residual error for , and is the nearest point to in , i.e.
It is intuitively clear that the symmetry plane fitting problem may also be solved via ICP, the only difference being the parameters over which the problem is solved (i.e. and ).
3.4 Transformed Symmetry Plane Parameters
Let us still assume that and are the symmetry plane parameters of a point set , and and are the parameters that align a point set with . Each transformed point and its transformed, symmetric equivalent must still fulfill the original reflection equation (3):
Cancelling and multiplying by on either side, we easily obtain
By comparing to (3), it is obvious that and must represent the symmetry plane parameters for the original, untransformed set .
4 Alignment and Symmetry as a Joint Optimization Problem
We now introduce our novel optimization objective which jointly optimizes an aligning point set transformation as well as the plane of symmetry. The objective is then solved in a branch-and-bound optimization paradigm, for which we introduce both the domain parameterization as well as the derivation of upper and lower bounds.
4.1 Objective Function
We still assume that our two partial object point sets are given by and , and that the symmetry plane is represented by the normal and depth . We define to be the corresponding reflected point set of . We furthermore define and to be the aligned sets in either direction. The symmetry fitting objective of employs
where is now chosen as the nearest neighbour from the set . The final registration error itself employs
where is chosen as the nearest neighbour from the set and weight is used to take the set as aligned points first. The overall objective function becomes
Direct optimization over , and using traditional ICP would easily get trapped in the nearest local minimum. We therefore propose to minimize the energy objective using the globally optimal branch-and-bound paradigm, an exhaustive search strategy that branches over the entire parameter space. In order to speed up the execution, the method derives upper and lower bounds for the minimal energy on each branch (i.e. sub-volume of the optimization space), and discards branches for which the lower bound remains higher than the upper bound in another branch. In the remainder of this section, we will discuss the two important questions of (i) how to parametrize and branch the parameter space, and (ii) how to find concrete values for the upper and lower bounds.
4.2 Domain Parameterization
2D problem: For a 2D point set, the domain of all feasible alignment parameters is represented by an angle to parameterize the planar rotation
, and a 2D translation vector. The space of all 2D rotations can therefore be compactly represented by the interval . For the translation, we assume that the optimal translation must lie within a cube defined by the interval . The symmetry plane in the 2D case becomes a line which is parameterized by an angle defining the normal vector , and a scale . Given the dual representation of line normal vectors, is constrained to the interval , and lies in the interval .
3D problem: The disadvantage of BnB is that its complexity grows exponentially in the dimensionality of the problem. We therefore take prior information about the 3D point sets into account that helps to decrease the dimensionality. We make the assumption that most objects are standing upright to the ground plane. The rotation between the different partial point sets is therefore still constrained to be a 1D rotation about the vertical axis, and the normal vector remains a 1D variable the horizontal plane. The translation however becomes a 3D vector over the interval .
In conclusion, the 2D alignment problem has 5 degrees of freedom, whereas the 3D problem turns into a-dimensional estimation problem.
4.3 Derivation of the Upper and Lower Bounds
The basic idea of BnB is to partition feasible set into convex sets which means it crucially depends on upper and lower bounds for the objective 2-energy on a given interval. Given that our objective (4.1) consists of a sum of many squared and positive sub-energies, lower and upper bounds on the overall objective energy can be derived by calculating individual upper and lower bounds on the alignment and symmetry residuals. Using the above introduced parametrizations, the upper bound and the lower bound of the optimal, joint registration and symmetry cost on a given interval of variables are therefore given as
Upper bounds on an interval are easily given by the energy of an arbitrary point within the interval. Given an interval centered at , upper bounds are therefore easily defined as
The remainder of this section discusses the derivation of the lower bounds.
Lower bound for the alignment error : For a rotation interval of half-length with centre , we have
is also called the rotation uncertainty radius.
Proof: Using Lemmas 3.1 and 3.2 of , we have:
We can similarly derive a translation uncertainty radius . For a translation volume with half side-length centered at , we have
Note that in the 2D case, we have . The lower bound of the registration term in equation (10) becomes
Lower bound of symmetry term : Assuming that the normal is defined by an -interval of half-length and with center , we have
For the depth , we simply have
Now let be the closest point to , and let be the closest point to . The lower bound is derived as follows:
We furthermore have
Similar to prior art , we improve the algorithm’s ability to handle the dimensionality of the problem by installing a nested BnB paradigm.
5.1 Nested BnB
We install a nested BnB scheme in which the outer layer searches through the space of all angular parameters, while the inner layer optimizes over the space of translation and depth. While finding the bounds in a sub-volume of the angle space, the algorithm calls the inner BnB algorithm to identify the optimal translation and scale. One important approximation that accelerates the execution is that—whenever the bounds in a sub-volume are derived—the uncertainty of the non-optimized parameters of that particular layer are set to zero. Detailed descriptions are given in Algorithm 1 (the Main Algorithm) and Algorithm 2 (the Inner BnB).
5.2 Integration with Local ICP
Within the outer layer, whenever BnB identifies an interval with an improved upper bound, we will execute a conventional local ICP algorithm starting from the center of the and taking and as an initial value. Once ICP converges to a local minimum with a lower function value, the new value is used to further reduce the upper bound. The technique is inspired by Yang et al. .
5.3 Trimming for Outlier Handling
A general problem with partially overlapping point sets is that—even at the global optimum—some points may simply not have a correspondence, and should hence be treated as outliers. Although the addition of symmetry and point reflections already greatly alleviates this problem, we still add the strategy proposed in Trimmed ICP  for robust point-set registration. More specifically, in each iteration, only a subset of the matched data points with smallest point-to-point distances are used for motion computation. In this work, we choose a -subset for both symmetry and registration residuals.
We now report our experimental results on both synthetic and real data. In all our experiments, we pre-normalized the pointsets such that all points are within the domain of . The parameter is set to (cf. Section 4.2
). We run experiments on both 2D and 3D data, and compare our results against the open-source implementation of Go-ICP complemented by the ransac-based symmetry detection method presented in and applied to a fusion of the aligned point sets. For Go-ICP, the stopping criterion is set to , and for our method it is set to where is the trimming ratio, are the number of points in and , respectively. For , the number of iterations is limited to .
6.1 Performance on 2D Synthetic Data
Each experiment is generated by taking an image that contains a symmetrical object, and using the Sobel edge detector to extract the object contour points. To evaluate the performance, we randomly divide the contour points into two subsets with a defined and structured overlap. To conclude, is transformed by a random rotation and translation drawn from the intervals degrees and .
Handling of limited overlap: The overlap between both point sets if varied from to . For each overlap ratio, we repeat 50 experiments each time choosing a random object, rotation and translation. Fig 1 shows an example result, where the left is the result of our proposed method, the centre is the result of the 2D version of Go-ICP, and the right one shows the ground truth alignment. Figure 2 shows all mean and median errors of all optimisation parameters over all evaluation results. As can be observed, our method has a substantially better ability to deal with partial overlaps compared to Go-ICP, thus leading to vastly improved symmetry plane parameters as well.
Outlier handling: To test resilience against outliers, we repeat the same experiment but add up to outliers to both and . Figure 3 indicates an example result, and Figure 4 again illustrates the mean and median errors over all experiments. While the registration error starts to increase earlier and the average angular errors tend to be higher, it can still be concluded that our method significantly outperforms Go-ICP followed by symmetry plane detection using .
6.2 Performance on 3D Synthetic Data
We choose symmetrical CAD models from ShapeNet , different types from classes. There objects contain more than one symmetry plane. For each object, we generate depth images (with occlusions) from random views around the object, and producing pairs of point sets for each object instance with varying overlap, and finally a total of about point-set registration experiments. Our result is shown in Figure 5, which again illustrates mean and median errors of all estimated quantities as a function of the overlap between the sets. In 3D, it is natural that the camera captures only a small part of the object, and that in turn even the fusion of both aligned point scans may not enable a stable estimation of the symmetry plane, hence the slightly increased mean error. However, especially the median error is still lower compared to Go-ICP, thus confirming the mutual benefits of our joint estimation paradigm. Figure 6 shows a few concrete examples for which the 3D objects only contain a single symmetry plane and for which our method is largely outperformed. Figure 7 shows failure examples where the partial point sets lead to an ambiguity in the symmetry planes. In particular, in the first example, Go-ICP still works as the overlap ratio is sufficient for the registration.
6.3 Experiments on Real Data
Our last experiment is an exciting application to real data that goes back to the initial motivation in the introduction. Figure 8 shows two different depth images captured by a Kinect camera, each one containing three instances of the same object under different orientations. By pairwise alignment of partial object scans, the mutual information is transferred thus leading to more complete perception of each individual object. Note that we use simple ground plane fitting and depth discontinuity-aware point clustering within object bounding boxes to isolate the partial object scans. With known position of the ground plane, we then transform the whole scene to be orthogonal to the ground plane and meet the assumption that all objects are placed upright and—in terms of relative rotation—differ only by an angle about the vertical axis. In Figure 8, the first column shows the original scan in different orientations, the second one the partial object measurements, the third one the completion obtained by using Go-ICP as an alignment algorithm and ransac-based symmetry detection, and the last one the result obtained by using our algorithm. As can be observed, our joint alignment strategy outperforms the comparison method, and achieves meaningful shape completion.
Symmetry detection and point set alignment over sets with small overlap are challenging problems if handled separately. Our work demonstrates a substantial improvement in both accuracy and success rate of the alignment by solving those two problems jointly. The information gained from estimating symmetry and reflecting points notably makes up for otherwise missing correspondences. However, our current implementation is not competitive in terms of running time, hence we are working on a parallel implementation. We furthermore plan to extend the algorithm to multi-point set registration, and improve its ability to deal with the situation of multiple symmetry planes.
-  L. Guibas P. Hanrahan Q. Huang Z. Li S. Savarese M. Savva S. Song H. Su et al. A. X. Chang, T. Funkhouser. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
-  K.S. Arun, T.S. Huang, and S.D. Blostein. Least-Squares Fitting of Two 3-D Point Sets. Transations of Pattern Analysis and Machine Intelligence, 9(5):698–700, 1987.
-  Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586–606. International Society for Optics and Photonics, 1992.
Dylan Campbell and Lars Petersson.
Gogma: Globally-optimal gaussian mixture alignment.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5685–5694, 2016.
-  Jia Chen, Xiaojun Wu, Michael Yu Wang, and Xuanfu Li. 3d shape modeling using a self-developed hand-held 3d laser scanner and an efficient ht-icp point cloud registration algorithm. Optics & Laser Technology, 45:414–423, 2013.
-  Yang Chen and Gérard Medioni. Object modelling by registration of multiple range images. Image and vision computing, 10(3):145–155, 1992.
-  Dmitry Chetverikov, Dmitry Stepanov, and Pavel Krsek. Robust euclidean alignment of 3d point sets: the trimmed iterative closest point algorithm. Image and vision computing, 23(3):299–309, 2005.
-  Haili Chui and Anand Rangarajan. A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding, 89(2-3):114–141, 2003.
-  Marcelo Cicconet, David GC Hildebrand, and Hunter Elliott. Finding mirror symmetry via registration and optimal symmetric pairwise assignment of curves: Algorithm and results. In Proceedings of the IEEE International Conference on Computer Vision, pages 1759–1763, 2017.
-  Andrea Cohen, Christopher Zach, Sudipta N Sinha, and Marc Pollefeys. Discovering and exploiting 3d symmetries in structure from motion. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1514–1521. IEEE, 2012.
-  Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381–395, 1981.
-  Sébastien Granger and Xavier Pennec. Multi-scale em-icp: A fast and robust approach for surface registration. In European Conference on Computer Vision, pages 418–432. Springer, 2002.
-  Richard I Hartley and Fredrik Kahl. Global optimization through rotation space search. International Journal of Computer Vision, 82(1):64–79, 2009.
-  M. Bloesch A. Davison J. McCormac, R. Clark and S. Leutenegger. Fusion++: Volumetric object-level slam. In In IEEE International Conference on 3D Vision (3DV), year = 2018.
-  Bing Jian and Baba C Vemuri. Robust point set registration using gaussian mixture models. IEEE transactions on pattern analysis and machine intelligence, 33(8):1633–1645, 2010.
-  Wei Jiang, Kai Xu, Zhi-Quan Cheng, and Hao Zhang. Skeleton-based intrinsic symmetry detection on point clouds. Graphical Models, 75(4):177–188, 2013.
-  Hongdong Li and Richard Hartley. The 3d-3d registration problem revisited. In 2007 IEEE 11th international conference on computer vision, pages 1–8. IEEE, 2007.
-  Yanxi Liu, Hagit Hel-Or, Craig S Kaplan, Luc Van Gool, et al. Computational symmetry in computer vision and computer graphics. Foundations and Trends® in Computer Graphics and Vision, 5(1–2):1–195, 2010.
-  Andriy Myronenko and Xubo Song. Point set registration: Coherent point drift. IEEE transactions on pattern analysis and machine intelligence, 32(12):2262–2275, 2010.
-  R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In International Symposium on Mixed and Augmented Reality, 2011.
-  Carl Olsson, Fredrik Kahl, and Magnus Oskarsson. Branch-and-bound methods for euclidean registration problems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5):783–794, 2008.
-  Alvaro Parra Bustos, Tat-Jun Chin, and David Suter. Fast rotation search with stereographic projections for 3d registration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3930–3937, 2014.
-  Kari Pulli. Multiview registration for large data sets. In Second International Conference on 3-D Digital Imaging and Modeling (Cat. No. PR00062), pages 160–168. IEEE, 1999.
-  Yanghai Tsin and Takeo Kanade. A correlation-based approach to robust point set registration. In European conference on computer vision, pages 558–569. Springer, 2004.
-  Kai Xu, Hao Zhang, Andrea Tagliasacchi, Ligang Liu, Guo Li, Min Meng, and Yueshan Xiong. Partial intrinsic reflectional symmetry of 3d shapes. ACM Transactions on Graphics (TOG), 28(5):138, 2009.
-  Jiaolong Yang, Hongdong Li, Dylan Campbell, and Yunde Jia. Go-icp: A globally optimal solution to 3d icp point-set registration. IEEE transactions on pattern analysis and machine intelligence, 38(11):2241–2254, 2015.
-  Jiaolong Yang, Hongdong Li, and Yunde Jia. Go-icp: Solving 3d registration efficiently and globally optimally. In Proceedings of the IEEE International Conference on Computer Vision, pages 1457–1464, 2013.
-  Zhengyou Zhang. Iterative point matching for registration of free-form curves and surfaces. International journal of computer vision, 13(2):119–152, 1994.