 # A Practical Maximum Clique Algorithm for Matching with Pairwise Constraints

A popular paradigm for 3D point cloud registration is by extracting 3D keypoint correspondences, then estimating the registration function from the correspondences using a robust algorithm. However, many existing 3D keypoint techniques tend to produce large proportions of erroneous correspondences or outliers, which significantly increases the cost of robust estimation. An alternative approach is to directly search for the subset of correspondences that are pairwise consistent, without optimising the registration function. This gives rise to the combinatorial problem of matching with pairwise constraints. In this paper, we propose a very efficient maximum clique algorithm to solve matching with pairwise constraints. Our technique combines tree searching with efficient bounding and pruning based on graph colouring. We demonstrate that, despite the theoretical intractability, many real problem instances can be solved exactly and quickly (seconds to minutes) with our algorithm, which makes our approach an excellent alternative to standard robust techniques for 3D registration.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

The registration of discrete 3D point sets or point clouds is a recurrent task in computer vision. Often, the point clouds correspond to objects or surfaces that were acquired from the environment using a 3D scanner. Registering point clouds allows to identify the parts in the point clouds that are “similar”. Thus, point cloud registration plays a major role in various applications, such as object recognition, robotic navigation, and digital reconstruction.

A popular paradigm for registering 3D point clouds is by extracting 3D keypoint correspondences, then estimating the registration function from the correspondences. Specifically, let and be two input point clouds. A 3D keypoint technique  is used to generate a tentative correspondence set , where each associates a point to a point . If there are no false correspondences or outliers, the registration function, i.e., a 6 DoF rigid transformation, can be estimated easily from  . Usually, however, outliers exist in , thus the registration function must be estimated using a robust technique  such as RANSAC .

Many current 3D keypoint techniques are in fact much less accurate than their 2D image counterparts [12, 11], since the irregular sampling densities on discrete point sets reduce the efficacy of local features. On real point clouds, it is common to encounter outlier rates in excess of . Such high outlier rates greatly increase the computational cost of robust estimation. It is thus vital to investigate alternative approaches for keypoint-based 3D registration.

Instead of estimating the registration function from , we can attempt to find the largest subset of that are pairwise consistent, i.e., we aim to solve

 maximiseI⊆{1,…,N}|I|subject tod(ci,cj)≤ϵ,∀i,j∈I, (1)

where the “distance” between two correspondences and is given by

 d(ci,cj)=∥∥(xi−xj)−(yi′−yj′)∥∥2. (2)

We say that and are consistent if is less than a predetermined threshold ; intuitively, this means that and are agreeable (up to ) to the same rigid transformation. Solving (1) then yields the largest subset of whose elements are all pairwise consistent; see Fig. 1.

Problem (1) is a special case of matching with pairwise constraints [1, 6, 16, 3]222In the general case, is taken as all pairings of and .. Observe that (1) does not involve optimising the registration function. Given the solution , the registration function can be estimated from the data indexed by (using RANSAC  or SVD  directly), or the value can directly be taken as the similarity score of shapes and . Theoretically, however, problem (1) is intractable (NP-hard) in general; as we will show in Sec. 2, (1) is expressible as the classical combinatorial problem of minimum vertex cover. The state-of-the-art algorithm in computer vision based on vertex cover  can handle only small instances of (1).

We note also that, whilst similar, problem (1) is not equivalent to the graph matching or quadratic assignment problem (QAP) [8, 18], since (1) explicitly forbids correspondences that are pairwise inconsistent. Expressing (1) as a QAP would involve a weight matrix with values, which poses significant difficulties for most QAP solvers.

#### Contributions

We show that problem (1) can be solved much more efficiently than previously thought. Specifically, we reformulate (1) as maximum clique, then propose a very efficient algorithm for the problem. Our technique is inspired by a state-of-the-art branch-and-bound (BnB) algorithm , but with important innovations to the bounding and pruning step based on graph colouring to significantly speed-up the solution - as we will show in Sec. 5, our algorithmic improvements are vital to enable the technique on practical-sized input data. Compared to the state-of-the-art method based on vertex cover , our method is one order of magnitude faster, and typically requires only seconds to minutes to globally solve (1) on realistic input data.

Our work is the first to illustrate matching with pairwise constraints as a strong alternative to conventional robust fitting procedures for keypoint-based 3D registration. Please see demo program in the supplementary material.

## 2 Graph formulation

Let represent an undirected graph with vertices and edges .

###### Definition 2.1 (Adjacency and degree).

We say that a pair of vertices and of are adjacent if . For each , denote the adjacency of as

 Γ(vi)={vj∈V|(vi,vj)∈E}. (3)

Then is called the degree of .

###### Definition 2.2 (Consistency graph).

Given a set of correspondences , the consistency graph is constructed as the graph with vertices and edges

 E={(ci,cj)∈C×C∣d(ci,cj)≤ϵ,i≠j}, (4)

i.e., two correspondences and are adjacent in the graph if they are pairwise consistent.

###### Definition 2.3 (Inconsistency graph).

The inconsistency graph is the complement of the consistency graph, i.e., the graph with vertices and edges

 E={(ci,cj)∈C×C∣d(ci,cj)>ϵ,i≠j}. (5)

i.e., two correspondences and are adjacent in the graph if they are pairwise inconsistent.

###### Definition 2.4 (Clique).

A clique of a graph is a subgraph of where every pair of vertices in the subgraph are adjacent. A maximum clique (MC) of is a clique of with the largest size.

###### Definition 2.5 (Vertex cover).

A vertex cover of a graph is a subset of such that every edge in is incident with at least one vertex in the subset. The removal of a vertex cover from leaves an independent set, i.e., a set of vertices with no edges. A minimum vertex cover (MVC) of is a vertex cover of with the smallest size.

By the above definitions, problem (1) is equivalent to finding the MC of the consistency graph constructed from the correspondence set . Conversely, the complementary problem to (1) is then finding the MVC of the inconsistency graph constructed from , i.e., remove the least number of correspondences such that the remaining correspondences are not pairwise inconsistent with each other.

### 2.1 MIP solutions

Both MC and MVC can be expressed as mixed integer programs (MIP). Given input graph , MC can be written as the MIP

 maximise|V|∑i=1xisubject toxi+xj≤1,∀(vi,vj)∉Exi∈{0,1},i=1…|V|. (6)

The MIP formulation for MVC is

 minimise|V|∑i=1xisubject toxi+xj≥1,∀(vi,vj)∈Exi∈{0,1},i=1…|V|. (7)

Both formulations can then be solved using industry-grade MIP solvers such as IBM CPLEX and Gurobi Optimiser. Since these are mature implementations, we do not further discuss their details; suffice to say that we will compare against the MIP solvers as baselines in the experiments.

For MVC, Enqvist et al.  proposed a BnB method where the bounding is conducted using a factor- approximation technique. Our own experimentation suggests however that their algorithm is generally slower than MIP. In Sec. 4, we propose a novel MC algorithm that is able to significantly outperform the generic MIP solvers.

## 3 Maximum clique algorithm

In this section, we first describe a state-of-the-art MC algorithm called MCQ . As we will show in Sec. 5, on realistic input data for point cloud registration, MCQ is still unable to provide good performance. However, MCQ will form the basis for our algorithm to be described in Sec. 4.

### 3.1 BnB

MCQ is a BnB algorithm. Given an input graph , BnB systematically explores the set of cliques of by building a search tree over the vertices . Before elaborating MCQ, we explain a basic BnB method summarised in Algorithm 1, which represents the basic structure of MCQ.

To more intuitively explain Algorithm 1, we use the sample input graph in Fig. 2. The first 12 steps of Algorithm 1 when applied to the sample graph are shown in Fig. 3.

In the initialisation, the set containing all vertices in forms the root node of the BnB search tree; see Step 1 in Fig. 3. The tree is then explored in a depth-first manner, where each branch is initiated by recursively expanding a vertex in . A vertex is expanded by creating a child node for that contains the vertices in that are adjacent to the vertex to be expanded. Observe in Step 2 in Fig. 3 that vertex in the root node is expanded to yield . The process then continues recursively - e.g., vertex in is expanded to yield node , and in is expanded to . Note that although is adjacent to other vertices (i.e., and ) in the graph, contains only since only exists in the candidate set from which was expanded.

The above recursive expansion strategy ensures that a candidate clique is associated with every node of the search tree ( for the root node) - e.g., is associated to , to , and to ; see Fig. 3.

When an empty node is reached, the largest clique found thus far is recorded as - e.g., in Step 4 in Fig. 3, the expansion of the single vertex of creates and is upgraded to .

After a complete branch is expanded for a vertex, that vertex is removed from the node - e.g., in Step 5 in Fig. 3, is eliminated from ; in Step 6, is eliminated from ; and in Step 8, is eliminated from the root node. This removing step does not compromise optimality since any larger clique containing and the removed vertex was already expanded.

A fundamental aspect of BnB algorithms is to prune branches in the search tree that are not promising. To conduct pruning, BnB evaluates an upper bound function of quality achievable at a branch. In Algorithm 1, we simply discard a branch if the depth of the node (equal to ) plus the number of nodes in the candidate set is not greater than the size of . In Step 7 in Fig. 3, the algorithm stops expanding vertices in since at most a clique of size 3 can be obtained by expanding its vertices (at that stage , and ).

### 3.2 Pruning with graph colouring

In the “full version” of MCQ, a more aggressive pruning is conducted using graph colouring. Algorithm 2 summarises the procedure.

A colouring of a graph is a labelling of its vertices such that no adjacent vertices have the same colour

 f(vi)≠f(vj)for all(vi,vj)∈E, (8)

and only consecutive numbers starting from 1 are used as colours. The last constraint ensures that the number of colours is equal to the largest colour.

To colour a graph containing a clique , at least colours are needed, since each vertex in is adjacent to vertices. Hence, a colouring of a subgraph in the search tree, can be used to obtain an upper bound of the size of the largest clique of expanding a vertex in .

 u=|R|+max{f(vi)|vi∈S}, (9)

where is the associated clique to .

To make a tight upper bound, and hence produce a more aggressive pruning in BnB, we want to find the colouring with the minimum number of colours

 minfmaxvi∈Sf(vi). (10)

Finding the minimum colouring (10

) is a well known NP-hard problem. MCQ uses a greedy heuristic algorithm for colouring: after an initial sorting of vertices that reduces the average branching factor (Line

3), vertices are reordered in increasing colour order such that the rightmost vertex in has the largest colour. Thus, vertices in are expanded and removed from right-to-left.

## 4 A faster maximum clique algorithm

To speed up MCQ, we introduce a novel extra pruning step to avoid exploring multiple branches during the search for the optimal solution; the resultant method is summarised in Algorithm 3. Our pruning step complements the original pruning technique based on graph colouring. We call the proposed algorithm PMC (Practical Maximum Clique).

To more intuitively explain the proposed pruning step, we again use the sample input graph in Fig. 2. The first 7 steps of Algorithm 3 until the optimal solution is reached are shown in Fig. 4 (contrast with the 12 steps needed for Algorithm 1; see Fig. 3). Vertices were expanded in the same order as in Algorithm 1 (Line 3 in Algorithm 3 was set to reorder vertices such that vertices are expanded in the order ).

For each node in the search tree of Algorithm 3, we aim to find a subset , such that only vertices in need to be expanded to find the optimal solution. In Fig. 4, , and .

Assume momentarily that does not change during the progression of Algorithm 3, i.e., no vertices are removed after expansions. Also assume that for a vertex in , all larger cliques expanded from have already been explored. If an unexpanded vertex in is adjacent to , any clique containing must have already been reached through expanding . Thus, there is no need to expand . This observation explains the core insight used by the proposed pruning.

Similar to Algorithm 1 and Algorithm 2, Algorithm 3 removes a vertex from after exploring its child nodes. To choose for a node , Algorithm 3 keeps record of removed vertices by adding them to a set associated to - e.g., in Step 5 in Fig. 4, is added to after it is removed from . In the root node of the search tree, is initiated as . For a child node , the corresponding set of removed vertices is initiated as

 F′=S∩F. (11)

can be obtained before expanding any vertex in . Note that must be in as only nodes in that are adjacent to will not be expanded (). These observations are formalised in the following theorem.

###### Theorem 1.

For a vertex in , the largest clique containing must contain at least one vertex in .

###### Proof.

The proof is by contradiction. Consider that the largest clique containing has been already obtained. Then, can be found by expanding vertices in . If with , then is a larger clique that contains . However, this contradicts being the largest clique. ∎

From Theorem 1, we can take , for any in . We choose such that we can skip expanding as many vertices as possible

 maximisevk∈S∪F|S∩Γ(vk)|. (12)

Solving (12) is equivalent to find the vertex with the highest degree in the subgraph with vertices in , which can be solved in quadratic time. Since (12) is solved only once in each node of the search tree, this extra step introduces minor computation cost. Experiments in Sec. 5 show that this extra step produces significant speed-ups when finding the optimal solution for problem (1).

Note that Algorithm 3 also uses pruning with graph colouring. The colouring of a node is obtained using the greedy heuristic used in MCQ. However, removing vertices from may invalidate that colours are consecutive numbers, since not necessarily the last vertex in (associated with the largest colour) is removed if expanding vertices in only. As colours are sorted, colour numbers can be reassigned in linear time (Line 24 in Algorithm 3). Figure 5: Qualitative results. Column 1: Input correspondence set C with N=3000. Column 2: Largest subset of pairwise consistent correspondences I∗. Column 3: The alignment given by the rigid transformation estimated from I∗ using SVD .

## 5 Results

To investigate the efficacy of the proposed method, we compared the following algorithms for matching with pairwise constraints (1):

• [itemsep=0.0pt,topsep=0.5pt,parsep=0.5pt]

• MCQ: BnB method (Algorithm 2) presented in  (using the C++ implementation of ).

• PMC: Our BnB method (Algorithm 3) with the same initial vertex ordering as MCQ.

• MIP-MC: MIP formulation of MC solved with Gurobi Optimiser.

• MIP-MVC: MIP formulation of MVC solved with Gurobi Optimiser.

PMC was implemented in C++, and all experiments were conducted on a standard PC with a 2.50 GHz CPU and 8 GB of RAM. Note that the currently state-of-the-art algorithm for matching with pairwise constraints in computer vision is . Our experimentations suggest however that Enqvist et al. (which is an MVC solver444Based on our own implementation. The original authors’ implementation was unavailable.) is outperformed generally by MIP-MVC, e.g., the former was unable to terminate within the imposed time limit on all data instances. We thus do not include Enqvist et al. in our benchmark.

We tested all methods on scans of objects from two different datasets, namely, the Stanford 3D Scanning Repository  (armadillo, buddha, bunny, and dragon), and Mian’s dataset  (chef, chicken, parasauro and t-rex). The scans of the objects are illustrated in Fig. 5.

Two partially overlapping scans and were selected for each object. and were centred and scaled such that both point sets were contained in the cube . The set of point correspondences was obtained from the sets of keypoints and using the state-of-the-art ISS3D  detector and the PFH  descriptor, as implemented on Point Cloud Library. The sizes of and are listed on the Column 1 in Table 1.

The best keypoint correspondences (sorted based on the -norm of the PFH descriptors), where , were retained to create instances of the pairwise maximisation problem (1) for point cloud registration. Based on the graph construction methods outlined in Sec. 2, the size of the consistency and the inconsistency graphs are listed in Columns 4–7 in Table 1. The inlier threshold for (1) was taken as twice the average nearest neighbour distance in and .

To give an indication of the ratio of correct correspondences generated for each and pair, we registered the point clouds using the ground truth rigid transformation and determined the subset of (the outliers) that cannot be aligned up to (using ensures the equivalence of the thresholds for pairwise consistent alignment (1) and “unary” alignment; see  for details). The outlier ratio for each input instance is listed in Column 3 in Table 1. Notice that the outlier rates are extremely high (up to ).

We recorded the following measures for each method:

• [itemsep=0.0pt,topsep=0.5pt,parsep=0.5pt]

• : The optimal value of (1).

• time (s): runtime of solving (1), i.e., after generating the input graph. A timeout of hour ( seconds) was imposed on all methods.

If a method could not terminate successfully within the time limit, the result was marked with a ‘-’ in the table. Note that since all the methods return the global solution to (1), they have the same solution quality and differ only in runtime. To assess the goodness of the registration on each pair of point clouds, a rigid transformation was estimated using SVD  on . The quality of the rigid transformation parameters was measured based on the following errors:

• [itemsep=0pt,topsep=0.5pt,parsep=0.5pt]

• angErr (°): Angular error of estimated rotation w.r.t. the true rotation.

• trErr: Translation error of estimated translation w.r.t. the true translation.

Columns 8–14 in Table 1 show the recorded measures.

Although in general MIP-MC reported lower runtimes than MIP-MVC, MIP-MC still is impractical, i.e., it was unable to terminate within 1 hour on half of the generated input instances. On the instances where it was able to finish within the time limit, MIP-MC took several orders of magnitude more time than PMC to find the optimal solution.

In general, PMC found the optimal solution in less than 10 seconds. Although MCQ performed better than PMC for , PMC converged considerably faster than MCQ for - e.g., the runtime of MCQ was more than 1000 seconds in armadillo and t-rex. Note that MCQ did not converge within the time limit for all instances. Since MCQ and PMC use the same greedy heuristic for colouring, the observed runtimes suggest that the proposed additional pruning step in Sec. 4 played an important role for solving (1) on large graphs. Observe that for , the number of vertices of the consistency graph is one order of magnitude larger than for . Note that we are more likely to obtain good registration from larger correspondence sets. Observe that for , the optimal solution of (1) did not produce a good alignment in all cases; in particular, the angular error was considerable for bunny and chicken. Thus, the performance gain of PMC over MCQ on large graphs is of high practical significance.

Fig. 5 depicts qualitative results for all the tested objects with correspondences. As alluded above, the rigid transformations were estimated using SVD from .

#### Comparison against RANSAC

As a baseline, on each input instance , we executed RANSAC  to find the largest subset of correspondences that are “unary” consistent, i.e., the largest consensus set. We used a confidence level for the stopping criterion. Column 15 in Table 1 shows the median runtime of RANSAC over 100 runs. Observe that the runtime of RANSAC is quite significant, i.e., up to one order of magnitude greater than the runtime of PMC. This is because the runtime of RANSAC grows exponentially with the outlier ratio, and all the instances in Table 1 have more than outliers.

## 6 Conclusions

We have shown that matching with pairwise constraints can be performed in reasonable time when posing the problem as maximum clique. We have also proposed a maximum clique algorithm that combines graph colouring with a proposed extra pruning step to very efficiently solve maximum clique. The obtained results demonstrate that, using the proposed algorithm, matching with pairwise constraints is a very practical approach for point cloud registration and an excellent alternative to standard robust estimation.

## References

•  R. Bolles and R. Cain. Recognizing and locating partially visible objects: The local-feature-focus method. The International Journal of Robotics Research, 1(3):57–82, Sept. 1982.
•  B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Computer Graphics (SIGGRAPH 1996 Proceedings), pages 303–312. ACM, 1996.
•  O. Enqvist, K. Josephson, and F. Kahl. Optimal correspondences from pairwise constraints. In ICCV, pages 1295–1302. IEEE, 2009.
•  D. Eppstein and D. Strash. Listing all maximal cliques in large sparse real-world graphs. In Experimental Algorithms, volume 6630, pages 364–375. Springer, Berlin, Heidelberg, 2011.
•  M. Fischler and R. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381–395, June 1981.
•  W. Grimson. Object Recognition by Computer: The Role of Geometric Constraints. Artificial intelligence. MIT Press, Cambridge, Mass, 1990.
•  B. Horn. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Am. A, 4(4):629–642, Apr. 1987.
•  M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints. In ICCV, volume 2, pages 1482–1489, Beijing, 2005. IEEE.
•  P. Meer. Robust techniques for computer vision. In Emerging Topics in Computer Vision, chapter 3, pages 107–190. Prentice Hall, 2004.
•  A. S. Mian, M. Bennamoun, and R. Owens. A novel representation and feature matching algorithm for automatic pairwise registration of range images. IJCV, 66(1):19–40, 2006.
•  K. Mikolajczyk, B. Leibe, and B. Schiele. Local features for object class recognition. In ICCV, pages 1792–1799, 2005.
•  K. Mikolajczyk and C. Schmid. Scale & affine invariant interest point detectors. IJCV, 60(1):63–86, Oct. 2004.
•  R. Rusu, N. Blodow, Z. Marton, and M. Beetz. Aligning point cloud views using persistent feature histograms. In International Conference on Intelligent Robots and Systems (IROS), pages 3384–3391, Nice, Sept. 2008. IEEE.
•  F. Tombari, S. Salti, and L. Di Stefano. Performance evaluation of 3D keypoint detectors. IJCV, 102(1-3):198–220, Mar. 2013.
•  E. Tomita and T. Seki. An efficient branch-and-bound algorithm for finding a maximum clique. In Discrete Mathematics and Theoretical Computer Science, volume 2731, pages 278–289. Springer, Berlin, Heidelberg, 2003.
•  P. Tu, R. Hartley, and T. Saxena. Recognizing objects using color-annotated adjacency graphs. In Shape, Contour and Grouping in Computer Vision, volume 1681, pages 246–263. Springer, Berlin, Heidelberg, 1999.
•  Y. Zhong. Intrinsic shape signatures: A shape descriptor for 3D object recognition. In ICCV Workshops, pages 689–696, Kyoto, 2009. IEEE.
•  F. Zhou and F. De la Torre. Deformable graph matching. In CVPR, pages 2922–2929. IEEE, June 2013.