1 Introduction
The registration of discrete 3D point sets or point clouds is a recurrent task in computer vision. Often, the point clouds correspond to objects or surfaces that were acquired from the environment using a 3D scanner. Registering point clouds allows to identify the parts in the point clouds that are “similar”. Thus, point cloud registration plays a major role in various applications, such as object recognition, robotic navigation, and digital reconstruction.
A popular paradigm for registering 3D point clouds is by extracting 3D keypoint correspondences, then estimating the registration function from the correspondences. Specifically, let and be two input point clouds. A 3D keypoint technique [14] is used to generate a tentative correspondence set , where each associates a point to a point . If there are no false correspondences or outliers, the registration function, i.e., a 6 DoF rigid transformation, can be estimated easily from [7]. Usually, however, outliers exist in , thus the registration function must be estimated using a robust technique [9] such as RANSAC [5].
Many current 3D keypoint techniques are in fact much less accurate than their 2D image counterparts [12, 11], since the irregular sampling densities on discrete point sets reduce the efficacy of local features. On real point clouds, it is common to encounter outlier rates in excess of . Such high outlier rates greatly increase the computational cost of robust estimation. It is thus vital to investigate alternative approaches for keypointbased 3D registration.
Instead of estimating the registration function from , we can attempt to find the largest subset of that are pairwise consistent, i.e., we aim to solve
(1) 
where the “distance” between two correspondences and is given by
(2) 
We say that and are consistent if is less than a predetermined threshold ; intuitively, this means that and are agreeable (up to ) to the same rigid transformation. Solving (1) then yields the largest subset of whose elements are all pairwise consistent; see Fig. 1.
Problem (1) is a special case of matching with pairwise constraints [1, 6, 16, 3]^{2}^{2}2In the general case, is taken as all pairings of and .. Observe that (1) does not involve optimising the registration function. Given the solution , the registration function can be estimated from the data indexed by (using RANSAC [5] or SVD [7] directly), or the value can directly be taken as the similarity score of shapes and . Theoretically, however, problem (1) is intractable (NPhard) in general; as we will show in Sec. 2, (1) is expressible as the classical combinatorial problem of minimum vertex cover. The stateoftheart algorithm in computer vision based on vertex cover [3] can handle only small instances of (1).
We note also that, whilst similar, problem (1) is not equivalent to the graph matching or quadratic assignment problem (QAP) [8, 18], since (1) explicitly forbids correspondences that are pairwise inconsistent. Expressing (1) as a QAP would involve a weight matrix with values, which poses significant difficulties for most QAP solvers.
Contributions
We show that problem (1) can be solved much more efficiently than previously thought. Specifically, we reformulate (1) as maximum clique, then propose a very efficient algorithm for the problem. Our technique is inspired by a stateoftheart branchandbound (BnB) algorithm [15], but with important innovations to the bounding and pruning step based on graph colouring to significantly speedup the solution  as we will show in Sec. 5, our algorithmic improvements are vital to enable the technique on practicalsized input data. Compared to the stateoftheart method based on vertex cover [3], our method is one order of magnitude faster, and typically requires only seconds to minutes to globally solve (1) on realistic input data.
Our work is the first to illustrate matching with pairwise constraints as a strong alternative to conventional robust fitting procedures for keypointbased 3D registration. Please see demo program in the supplementary material.
2 Graph formulation
Let represent an undirected graph with vertices and edges .
Definition 2.1 (Adjacency and degree).
We say that a pair of vertices and of are adjacent if . For each , denote the adjacency of as
(3) 
Then is called the degree of .
Definition 2.2 (Consistency graph).
Given a set of correspondences , the consistency graph is constructed as the graph with vertices and edges
(4) 
i.e., two correspondences and are adjacent in the graph if they are pairwise consistent.
Definition 2.3 (Inconsistency graph).
The inconsistency graph is the complement of the consistency graph, i.e., the graph with vertices and edges
(5) 
i.e., two correspondences and are adjacent in the graph if they are pairwise inconsistent.
Definition 2.4 (Clique).
A clique of a graph is a subgraph of where every pair of vertices in the subgraph are adjacent. A maximum clique (MC) of is a clique of with the largest size.
Definition 2.5 (Vertex cover).
A vertex cover of a graph is a subset of such that every edge in is incident with at least one vertex in the subset. The removal of a vertex cover from leaves an independent set, i.e., a set of vertices with no edges. A minimum vertex cover (MVC) of is a vertex cover of with the smallest size.
By the above definitions, problem (1) is equivalent to finding the MC of the consistency graph constructed from the correspondence set . Conversely, the complementary problem to (1) is then finding the MVC of the inconsistency graph constructed from , i.e., remove the least number of correspondences such that the remaining correspondences are not pairwise inconsistent with each other.
2.1 MIP solutions
Both MC and MVC can be expressed as mixed integer programs (MIP). Given input graph , MC can be written as the MIP
(6) 
The MIP formulation for MVC is
(7) 
Both formulations can then be solved using industrygrade MIP solvers such as IBM CPLEX and Gurobi Optimiser. Since these are mature implementations, we do not further discuss their details; suffice to say that we will compare against the MIP solvers as baselines in the experiments.
For MVC, Enqvist et al. [3] proposed a BnB method where the bounding is conducted using a factor approximation technique. Our own experimentation suggests however that their algorithm is generally slower than MIP. In Sec. 4, we propose a novel MC algorithm that is able to significantly outperform the generic MIP solvers.
3 Maximum clique algorithm
In this section, we first describe a stateoftheart MC algorithm called MCQ [15]. As we will show in Sec. 5, on realistic input data for point cloud registration, MCQ is still unable to provide good performance. However, MCQ will form the basis for our algorithm to be described in Sec. 4.
3.1 BnB
MCQ is a BnB algorithm. Given an input graph , BnB systematically explores the set of cliques of by building a search tree over the vertices . Before elaborating MCQ, we explain a basic BnB method summarised in Algorithm 1, which represents the basic structure of MCQ.
To more intuitively explain Algorithm 1, we use the sample input graph in Fig. 2. The first 12 steps of Algorithm 1 when applied to the sample graph are shown in Fig. 3.
In the initialisation, the set containing all vertices in forms the root node of the BnB search tree; see Step 1 in Fig. 3. The tree is then explored in a depthfirst manner, where each branch is initiated by recursively expanding a vertex in . A vertex is expanded by creating a child node for that contains the vertices in that are adjacent to the vertex to be expanded. Observe in Step 2 in Fig. 3 that vertex in the root node is expanded to yield . The process then continues recursively  e.g., vertex in is expanded to yield node , and in is expanded to . Note that although is adjacent to other vertices (i.e., and ) in the graph, contains only since only exists in the candidate set from which was expanded.
The above recursive expansion strategy ensures that a candidate clique is associated with every node of the search tree ( for the root node)  e.g., is associated to , to , and to ; see Fig. 3.
When an empty node is reached, the largest clique found thus far is recorded as  e.g., in Step 4 in Fig. 3, the expansion of the single vertex of creates and is upgraded to .
After a complete branch is expanded for a vertex, that vertex is removed from the node  e.g., in Step 5 in Fig. 3, is eliminated from ; in Step 6, is eliminated from ; and in Step 8, is eliminated from the root node. This removing step does not compromise optimality since any larger clique containing and the removed vertex was already expanded.
A fundamental aspect of BnB algorithms is to prune branches in the search tree that are not promising. To conduct pruning, BnB evaluates an upper bound function of quality achievable at a branch. In Algorithm 1, we simply discard a branch if the depth of the node (equal to ) plus the number of nodes in the candidate set is not greater than the size of . In Step 7 in Fig. 3, the algorithm stops expanding vertices in since at most a clique of size 3 can be obtained by expanding its vertices (at that stage , and ).
3.2 Pruning with graph colouring
In the “full version” of MCQ, a more aggressive pruning is conducted using graph colouring. Algorithm 2 summarises the procedure.
A colouring of a graph is a labelling of its vertices such that no adjacent vertices have the same colour
(8) 
and only consecutive numbers starting from 1 are used as colours. The last constraint ensures that the number of colours is equal to the largest colour.
To colour a graph containing a clique , at least colours are needed, since each vertex in is adjacent to vertices. Hence, a colouring of a subgraph in the search tree, can be used to obtain an upper bound of the size of the largest clique of expanding a vertex in .
(9) 
where is the associated clique to .
To make a tight upper bound, and hence produce a more aggressive pruning in BnB, we want to find the colouring with the minimum number of colours
(10) 
Finding the minimum colouring (10
) is a well known NPhard problem. MCQ uses a greedy heuristic algorithm for colouring: after an initial sorting of vertices that reduces the average branching factor (Line
3), vertices are reordered in increasing colour order such that the rightmost vertex in has the largest colour. Thus, vertices in are expanded and removed from righttoleft.4 A faster maximum clique algorithm
To speed up MCQ, we introduce a novel extra pruning step to avoid exploring multiple branches during the search for the optimal solution; the resultant method is summarised in Algorithm 3. Our pruning step complements the original pruning technique based on graph colouring. We call the proposed algorithm PMC (Practical Maximum Clique).
To more intuitively explain the proposed pruning step, we again use the sample input graph in Fig. 2. The first 7 steps of Algorithm 3 until the optimal solution is reached are shown in Fig. 4 (contrast with the 12 steps needed for Algorithm 1; see Fig. 3). Vertices were expanded in the same order as in Algorithm 1 (Line 3 in Algorithm 3 was set to reorder vertices such that vertices are expanded in the order ).
For each node in the search tree of Algorithm 3, we aim to find a subset , such that only vertices in need to be expanded to find the optimal solution. In Fig. 4, , and .
Assume momentarily that does not change during the progression of Algorithm 3, i.e., no vertices are removed after expansions. Also assume that for a vertex in , all larger cliques expanded from have already been explored. If an unexpanded vertex in is adjacent to , any clique containing must have already been reached through expanding . Thus, there is no need to expand . This observation explains the core insight used by the proposed pruning.
Similar to Algorithm 1 and Algorithm 2, Algorithm 3 removes a vertex from after exploring its child nodes. To choose for a node , Algorithm 3 keeps record of removed vertices by adding them to a set associated to  e.g., in Step 5 in Fig. 4, is added to after it is removed from . In the root node of the search tree, is initiated as . For a child node , the corresponding set of removed vertices is initiated as
(11) 
can be obtained before expanding any vertex in . Note that must be in as only nodes in that are adjacent to will not be expanded (). These observations are formalised in the following theorem.
Theorem 1.
For a vertex in , the largest clique containing must contain at least one vertex in .
Proof.
The proof is by contradiction. Consider that the largest clique containing has been already obtained. Then, can be found by expanding vertices in . If with , then is a larger clique that contains . However, this contradicts being the largest clique. ∎
From Theorem 1, we can take , for any in . We choose such that we can skip expanding as many vertices as possible
(12) 
Solving (12) is equivalent to find the vertex with the highest degree in the subgraph with vertices in , which can be solved in quadratic time. Since (12) is solved only once in each node of the search tree, this extra step introduces minor computation cost. Experiments in Sec. 5 show that this extra step produces significant speedups when finding the optimal solution for problem (1).
Note that Algorithm 3 also uses pruning with graph colouring. The colouring of a node is obtained using the greedy heuristic used in MCQ. However, removing vertices from may invalidate that colours are consecutive numbers, since not necessarily the last vertex in (associated with the largest colour) is removed if expanding vertices in only. As colours are sorted, colour numbers can be reassigned in linear time (Line 24 in Algorithm 3).
Object  Outlier  Consistency graph  Inconsistency graph  Solution  PMC  MCQ  MIPMC  MIPVC  RANSAC  

ratio  angErr (°)  trErr  time (s)  time (s)  time (s)  time (s)  time (s)  
armadillo  1000  0.98  1000  167200  1000  831800  37  0.72  0.06  0.020  0.010  1387.404  1291.205  25.25 
3000  0.98  3000  1397630  3000  7599370  92  0.61  0.05  2.510  5.390  –  –  39.93  
5000  0.98  5000  3485008  5000  21509992  139  0.98  0.15  61.080  3377.430  –  –  70.16  
buddha  1000  0.96  1000  139516  1000  859484  55  0.64  0.34  0.170  0.020  75.170  85.624  5.67 
3000  0.98  3000  700794  3000  8296206  68  1.60  0.18  5.420  2.730  –  –  105.24  
5000  0.99  5000  1700434  5000  23294566  74  1.06  0.25  5.640  76.100  –  –  602.04  
bunny  1000  0.96  1000  101956  1000  897044  27  2.03  0.42  0.020  0.010  –  –  5.09 
3000  0.96  3000  918160  3000  8078840  102  0.32  0.11  0.410  0.280  1224.189  1642.969  7.32  
5000  0.96  5000  2490092  5000  22504908  171  0.47  0.09  7.250  –  3520.973  –  16.39  
chef  1000  0.94  1000  117182  1000  881818  80  1.01  0.21  0.310  0.040  16.469  22.378  1.32 
3000  0.97  3000  774036  3000  8222964  97  0.95  0.16  1.980  1.410  1578.146  2293.338  30.39  
5000  0.98  5000  1961382  5000  23033618  100  0.61  0.26  6.870  37.470  –  –  200.01  
chicken  1000  0.97  1000  142856  1000  856144  26  15.88  2.02  0.040  0.020  –  –  14.53 
3000  0.98  3000  1265814  3000  7731186  55  1.63  0.28  1.480  0.830  –  –  58.08  
5000  0.98  5000  3597810  5000  21397190  81  0.93  0.23  7.620  21.420  –  –  98.33  
dragon  1000  0.91  1000  141516  1000  857484  106  0.23  0.20  0.090  0.020  19.240  21.579  0.52 
3000  0.97  3000  877756  3000  8119244  126  0.41  0.20  1.030  0.930  772.552  1294.577  12.74  
5000  0.98  5000  2211540  5000  22783460  136  0.31  0.19  5.530  18.130  –  –  76.86  
parasauro  1000  0.93  1000  153806  1000  845194  81  0.14  0.10  0.040  0.020  19.053  24.703  1.26 
3000  0.97  3000  973874  3000  8023126  118  0.40  0.10  2.830  42.160  2289.741  2681.053  21.83  
5000  0.98  5000  2214264  5000  22780736  127  0.44  0.22  36.600  –  –  –  84.86  
trex  1000  0.93  1000  116406  1000  882594  86  0.43  0.15  0.040  0.010  13.601  15.705  0.88 
3000  0.97  3000  818628  3000  8178372  118  0.13  0.21  1.200  10.570  865.289  1339.081  18.15  
5000  0.98  5000  2022928  5000  22972072  128  0.27  0.22  7.970  1287.740  –  –  81.86 
Input correspondence set  Pairwise consistent correspondences  Alignment  
armadillo 

buddha 

bunny 

chef 

chicken 

dragon 

parasauro 

trex 
5 Results
To investigate the efficacy of the proposed method, we compared the following algorithms for matching with pairwise constraints (1):

[itemsep=0.0pt,topsep=0.5pt,parsep=0.5pt]

MCQ: BnB method (Algorithm 2) presented in [15] (using the C++ implementation of [4]^{3}^{3}3https://github.com/darrenstrash/quickcliques).

PMC: Our BnB method (Algorithm 3) with the same initial vertex ordering as MCQ.

MIPMC: MIP formulation of MC solved with Gurobi Optimiser.

MIPMVC: MIP formulation of MVC solved with Gurobi Optimiser.
PMC was implemented in C++, and all experiments were conducted on a standard PC with a 2.50 GHz CPU and 8 GB of RAM. Note that the currently stateoftheart algorithm for matching with pairwise constraints in computer vision is [3]. Our experimentations suggest however that Enqvist et al. (which is an MVC solver^{4}^{4}4Based on our own implementation. The original authors’ implementation was unavailable.) is outperformed generally by MIPMVC, e.g., the former was unable to terminate within the imposed time limit on all data instances. We thus do not include Enqvist et al. in our benchmark.
We tested all methods on scans of objects from two different datasets, namely, the Stanford 3D Scanning Repository [2] (armadillo, buddha, bunny, and dragon), and Mian’s dataset [10] (chef, chicken, parasauro and trex). The scans of the objects are illustrated in Fig. 5.
Two partially overlapping scans and were selected for each object. and were centred and scaled such that both point sets were contained in the cube . The set of point correspondences was obtained from the sets of keypoints and using the stateoftheart ISS3D [17] detector and the PFH [13] descriptor, as implemented on Point Cloud Library^{5}^{5}5http://pointclouds.org/. The sizes of and are listed on the Column 1 in Table 1.
The best keypoint correspondences (sorted based on the norm of the PFH descriptors), where , were retained to create instances of the pairwise maximisation problem (1) for point cloud registration. Based on the graph construction methods outlined in Sec. 2, the size of the consistency and the inconsistency graphs are listed in Columns 4–7 in Table 1. The inlier threshold for (1) was taken as twice the average nearest neighbour distance in and .
To give an indication of the ratio of correct correspondences generated for each and pair, we registered the point clouds using the ground truth rigid transformation and determined the subset of (the outliers) that cannot be aligned up to (using ensures the equivalence of the thresholds for pairwise consistent alignment (1) and “unary” alignment; see [3] for details). The outlier ratio for each input instance is listed in Column 3 in Table 1. Notice that the outlier rates are extremely high (up to ).
We recorded the following measures for each method:

[itemsep=0.0pt,topsep=0.5pt,parsep=0.5pt]

: The optimal value of (1).

time (s): runtime of solving (1), i.e., after generating the input graph. A timeout of hour ( seconds) was imposed on all methods.
If a method could not terminate successfully within the time limit, the result was marked with a ‘’ in the table. Note that since all the methods return the global solution to (1), they have the same solution quality and differ only in runtime. To assess the goodness of the registration on each pair of point clouds, a rigid transformation was estimated using SVD [7] on . The quality of the rigid transformation parameters was measured based on the following errors:

[itemsep=0pt,topsep=0.5pt,parsep=0.5pt]

angErr (°): Angular error of estimated rotation w.r.t. the true rotation.

trErr: Translation error of estimated translation w.r.t. the true translation.
Columns 8–14 in Table 1 show the recorded measures.
Although in general MIPMC reported lower runtimes than MIPMVC, MIPMC still is impractical, i.e., it was unable to terminate within 1 hour on half of the generated input instances. On the instances where it was able to finish within the time limit, MIPMC took several orders of magnitude more time than PMC to find the optimal solution.
In general, PMC found the optimal solution in less than 10 seconds. Although MCQ performed better than PMC for , PMC converged considerably faster than MCQ for  e.g., the runtime of MCQ was more than 1000 seconds in armadillo and trex. Note that MCQ did not converge within the time limit for all instances. Since MCQ and PMC use the same greedy heuristic for colouring, the observed runtimes suggest that the proposed additional pruning step in Sec. 4 played an important role for solving (1) on large graphs. Observe that for , the number of vertices of the consistency graph is one order of magnitude larger than for . Note that we are more likely to obtain good registration from larger correspondence sets. Observe that for , the optimal solution of (1) did not produce a good alignment in all cases; in particular, the angular error was considerable for bunny and chicken. Thus, the performance gain of PMC over MCQ on large graphs is of high practical significance.
Fig. 5 depicts qualitative results for all the tested objects with correspondences. As alluded above, the rigid transformations were estimated using SVD from .
Comparison against RANSAC
As a baseline, on each input instance , we executed RANSAC [5] to find the largest subset of correspondences that are “unary” consistent, i.e., the largest consensus set. We used a confidence level for the stopping criterion. Column 15 in Table 1 shows the median runtime of RANSAC over 100 runs. Observe that the runtime of RANSAC is quite significant, i.e., up to one order of magnitude greater than the runtime of PMC. This is because the runtime of RANSAC grows exponentially with the outlier ratio, and all the instances in Table 1 have more than outliers.
6 Conclusions
We have shown that matching with pairwise constraints can be performed in reasonable time when posing the problem as maximum clique. We have also proposed a maximum clique algorithm that combines graph colouring with a proposed extra pruning step to very efficiently solve maximum clique. The obtained results demonstrate that, using the proposed algorithm, matching with pairwise constraints is a very practical approach for point cloud registration and an excellent alternative to standard robust estimation.
References
 [1] R. Bolles and R. Cain. Recognizing and locating partially visible objects: The localfeaturefocus method. The International Journal of Robotics Research, 1(3):57–82, Sept. 1982.
 [2] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Computer Graphics (SIGGRAPH 1996 Proceedings), pages 303–312. ACM, 1996.
 [3] O. Enqvist, K. Josephson, and F. Kahl. Optimal correspondences from pairwise constraints. In ICCV, pages 1295–1302. IEEE, 2009.
 [4] D. Eppstein and D. Strash. Listing all maximal cliques in large sparse realworld graphs. In Experimental Algorithms, volume 6630, pages 364–375. Springer, Berlin, Heidelberg, 2011.
 [5] M. Fischler and R. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381–395, June 1981.
 [6] W. Grimson. Object Recognition by Computer: The Role of Geometric Constraints. Artificial intelligence. MIT Press, Cambridge, Mass, 1990.
 [7] B. Horn. Closedform solution of absolute orientation using unit quaternions. J. Opt. Soc. Am. A, 4(4):629–642, Apr. 1987.
 [8] M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints. In ICCV, volume 2, pages 1482–1489, Beijing, 2005. IEEE.
 [9] P. Meer. Robust techniques for computer vision. In Emerging Topics in Computer Vision, chapter 3, pages 107–190. Prentice Hall, 2004.
 [10] A. S. Mian, M. Bennamoun, and R. Owens. A novel representation and feature matching algorithm for automatic pairwise registration of range images. IJCV, 66(1):19–40, 2006.
 [11] K. Mikolajczyk, B. Leibe, and B. Schiele. Local features for object class recognition. In ICCV, pages 1792–1799, 2005.
 [12] K. Mikolajczyk and C. Schmid. Scale & affine invariant interest point detectors. IJCV, 60(1):63–86, Oct. 2004.
 [13] R. Rusu, N. Blodow, Z. Marton, and M. Beetz. Aligning point cloud views using persistent feature histograms. In International Conference on Intelligent Robots and Systems (IROS), pages 3384–3391, Nice, Sept. 2008. IEEE.
 [14] F. Tombari, S. Salti, and L. Di Stefano. Performance evaluation of 3D keypoint detectors. IJCV, 102(13):198–220, Mar. 2013.
 [15] E. Tomita and T. Seki. An efficient branchandbound algorithm for finding a maximum clique. In Discrete Mathematics and Theoretical Computer Science, volume 2731, pages 278–289. Springer, Berlin, Heidelberg, 2003.
 [16] P. Tu, R. Hartley, and T. Saxena. Recognizing objects using colorannotated adjacency graphs. In Shape, Contour and Grouping in Computer Vision, volume 1681, pages 246–263. Springer, Berlin, Heidelberg, 1999.
 [17] Y. Zhong. Intrinsic shape signatures: A shape descriptor for 3D object recognition. In ICCV Workshops, pages 689–696, Kyoto, 2009. IEEE.
 [18] F. Zhou and F. De la Torre. Deformable graph matching. In CVPR, pages 2922–2929. IEEE, June 2013.
Comments
There are no comments yet.