The assignment problem finds an assignment, or matching, between two finite sets and , each of cardinality , such that the total cost of all matched pairs is minimized. The assignment problem can also be generalized to finding matchings between more than two sets. This is a fundamental problem in computer science and has been motivated by a wide gamut of research areas spanning diverse areas such as structural biology (Singer & Shkolnisky, 2011), protein structure comparisons in bioinformatics (Zaslavskiy et al., 2009)
, and computer vision(Conte et al., 2004). Computer vision especially boasts a broad range of applications that include object matching, image registration (Shen & Davatzikos, 2002), stereo matching (Goesele et al., 2007), shape matching (Petterson et al., 2009; Berg et al., 2005), structure from motion (SfM) (Szeliski, 2010), and object detection (Jiang et al., 2011), to name a few.
Various assignment approaches can broadly be classified as those that find a bijective assignment in the form of a permutation matrix by posing the problem as alinear assignment problem (LP) versus ones that solve a quadratic assignment problem (QAP) via graph matching, where each graph’s nodes represent the objects and the edges encode their corresponding distances; the goal of QAP then is to find node-wise correspondences between the graphs so that the overall discrepancy between their corresponding edge-wise counterparts is minimized and the overall relational structure is best preserved.
Partial assignment implies that only subsets of and can actually be assigned to each other successfully. This phenomenon is of particular interest to applications where either objects are absent due to incomplete observations, undergo deformations, and/or the objects in question cannot clearly be disambiguated because the objects in question along with their related objects are embedded in clutter. This variant of the assignment problem is widely accepted as a formidable challenge.
Although graph matching methods were found to be instrumental, they too perform poorly when faced with non-similar geometric transformations or transformations that produce degenerate triangulations. This is attributed to assigning weights to only node and edge assignments, while ignoring the interplay of higher-order connections/relations. For example, using triplet weights can alleviate the above mentioned problem to a very large extent by defining a measure invariant to scale and other transformations (Chertok & Keller, 2010).
Motivated by the aforementioned observations and inspired by Kahle (Kahle, 2006)’s work on combinatorial topological models like the random clique complex, we focus our attention to matching higher-order components between two sets of points in the setting of some points missing completely at random. We pose our assignment problem as finding a matching between two sets of points, each represented as a random clique complex, which is a higher-order analogue of random graphs. Figure 1 illustrates such a matching of cliques of corresponding dimensionality, between two different scenes (taken from different camera angles) of the same house. Given an Erdős-Rényi (ER) graph, its clique complex is the simplicial complex with all complete subgraphs (i.e., cliques) as its faces. The Erdős-Rényi graph forms the -skeleton of the random clique complex, where the cliques have at most a dimension of
, i.e., edges in the graph. The clique topology of a random adjacency matrix is analogous to its eigenvalue spectrum, as it provides a set of invariants that help detect structure(Giusti et al., 2015). This probabilistic and combinatorial framework of random clique complexes allows us to further study the assignment problem under various assumptions of the underlying distribution of the matrix entry distributions, its robustness to missing values, and its asymptotic behavior for large-scale cases.
Contributions: We present the following contributions.
To the best of our knowledge, our proposed approach is a first attempt to formulate higher-order matching between two sets of points, given partial or incomplete information, as a matching between two random clique complexes. We also propose an efficient matching algorithm and study both its time and storage complexity.
(i) We provide new bounds on the concentration inequality of eigenvalues of the QAP trace formulation for random symmetric matrices, (ii) we give tighter concentration inequality bounds on the largest eigenvalue for the Lawler QAP formulation on random matrices, in the context of affinity matrices
that are used by some earlier works. Furthermore, we theoretically analyze and discuss the robustness of affinity-matrix based schemes to missing points, and (iii) we perform asymptotic analysis on the worst to best case ratio of a QAP solution for our higher-dimensional clique adjacency matrices in theclique percolation regime (Bollobás & Riordan, 2009), where the entries follow a Poisson distribution.
Finally, we present a comprehensive empirical study that compares our method’s matching accuracy to that of a diverse set of matching approaches (Zhou & De la Torre, 2016; Zhou & De la Torre, 2013; Cho et al., 2010; Feizi et al., 2016; Leordeanu & Hebert, 2005; Cour et al., 2007; Pachauri et al., 2013; Gold & Rangarajan, 1996; Kuhn, 1955; Leordeanu et al., 2009; Zass & Shashua, 2008; Li et al., 2013; Duchenne et al., 2011). We conducted our experiments on both synthetic and well-known hard real-world datasets that span across affine/non-affine transformations, severe occlusions, and clutter. Our study reveals much better accuracy for the popular datasets against several of the state-of-the-art matching methods.
2 Matching Random Clique Complexes
We consider the problem of capturing higher-order feature groups among landmark points in an image by representing them as a random clique complex (RCC) and then using these RCCs to match two sets of groupings from two different images. We begin this section by describing the construction of a random clique complex, followed by our proposed method of matching two RCCs, and we finally analyze the runtime and storage complexity of our algorithm.
2.1 Structure of a Random Clique Complex
We begin with general definitions pertaining to the structure of simplicial complexes and then accordingly adapt these definitions to our domain of random graphs to build random clique complexes.
Let be an Erdős-Rényi graph with a set of vertices denoted by , whose edges , are i.i.d Bernoulli() distributed. Recall, that a -clique in is a complete subgraph that comprises of vertices and edges. Here onwards, for ease of notation, we will denote as . Given any affinely independent set of points in , the -simplex is the convex hull of , i.e., it is the set of all points of the form , where and for all . If we imagine the vertices of embedded generically in , then each -clique consisting of vertices is represented by a -dimensional simplex in our random clique complex. For example, a -clique (edge) and a -clique (triangle) in is represented as and , respectively.
Given , the -th face of is the subspace of points that satisfy ; it is the -simplex whose vertices are all those of , except the -th vertex. In other words, when is a clique of , then all its subsets are also cliques and hence considered faces of . For example, a -clique (triangle) has three -cliques (edges) in it.
With the aforementioned definitions in mind, we define our random clique complex as the set of all cliques in such that . We denote a set of -cliques as . Additionally, also satisfies the following conditions of a simplicial complex: (i) Any face in is also a simplex in and (ii) the intersection of any two simplexes is a face (lower dimensional clique) of both and .
The faces of are copies of for , which are glued together inductively. The -skeleton of , for , is defined as the following quotient space
where is the equivalence relation that identifies faces of to the corresponding faces of where . Finally, .
-skeleton as adjacency matrix: Given a random graph and its -skeleton that contains all its -cliques, we follow the idea from Bollobás et. al. (Bollobás & Riordan, 2009), to represent as an adjacency matrix whose vertex set is the set of of all - cliques in and in which two vertices (i.e., -cliques) are adjacent when they share a common face that has a minimum of vertices, where and . Such an adjacency matrix is built for each -skeleton and therefore is expressed as a set of matrices , where is the dimension of the cliques.
2.2 Problem Setup
The problem of matching random clique complexes each of dimension
, is the estimation of a set of optimal bijective maps of the form, for all , subject to assignment constraints. This can be formulated as a constrained quadratic assignment problem
, which can later be relaxed to a linear programming optimization problem.
Given two -dimensional random clique complexes and , let be a set of permutation matrices such that encodes assignments/matchings from to . The combinatorial matching requires the optimal set of permutation matrices that best align and . More formally, this can be expressed as the following constrained optimization problem
2.3 Our Algorithm
At a high level, our goal is to minimize , where is a combinatorial distance between two random clique complexes. Traditional metrics like Hausdorff distance are not suitable here because random clique complexes are combinatorial topological spaces. Recall that is comprised of a family of -skeletons , where each -skeleton contains cliques whose dimension is at most and has a maximum dimension . The solution of the optimization problem outlined in Equation (1) aims to find a set of permutation matrices that minimizes the overall number of misalignments between equi-dimensional faces of and , i.e., cliques belonging to the corresponding -skeletons, and thus producing the optimal least cost assignment between and .
presents our method to solve the combinatorial optimization problem (Equation (1)). In decreasing order of clique dimensionality, for a fixed dimension and given the adjacency matrices and for -skeletons and , respectively. In every iteration, our objective is to solve to find the optimal permutation . We assume the barycenters of every clique is pre-computed (Step ). Next, the neighborhood of the -th clique is computed as the set of entries with s in the -th row of (Step ). We denote the collection of every clique’s neighborhood as (Step ). An important objective of our method is to capture the geometric properties of the neighborhood of every clique. We achieve this by characterizing the -th clique’s barycenter as an affine combination of the barycenters (in all dimensions) associated with the cliques in its corresponding neighborhood . Given an arbitrary clique’s barycenter , let denote the barycenters of its adjacent cliques. Then, expressed as is an affine combination of the s, if , i.e., the weights sum to . Among all possible affine representations of we chose to use least squares to guarantee minimal error under L-norm, and furthermore it assigns non-zero weights to each of its adjacent clique barycenters, thereby capturing the local geometric properties in its neighborhood. The weight vector is then calculated for each clique (Step ) and denotes a collection of such weight vectors (Step ). Next, a cost matrix is built by computing the L-norm distance between weight vectors and (Step ). Finally, the Kuhn-Munkres (Kuhn, 1955) algorithm is invoked with both the adjacency matrices and the cost matrix, which arrives at the optimal assignment (Step ). At the end of all iterations, our method returns a set of optimal assignments for matches between each -skeleton for every dimension below and the algorithm terminates. We refer the reader to our supplementary section for a working example.
2.4 Complexity Analysis
To begin our analysis, we must first ascertain the dimensionality of , which is governed by the total number of -cliques that exist in the underlying random graph . It is important to note that there doesn’t exist any closed form solution to counting the number of cliques of a given dimension in .
We consider the distribution of a random variablecounting the number of -cliques in a realization of . We show in Appendix A of our supplementary material that this count is upper bounded by , where is Euler’s number. This can be expressed in asymptotic notation as . As dimensionality increases, there occurs an explosion in the number of cliques. Fortunately, is a sparse matrix and its effective dimensionality measured by the number of non-zero rows, i.e., the number of cliques with non-empty neighborhoods, is of order . Therefore, we set out to count the number of non-zero entries in .
We use a seminal result by Bollobás (Bollobás & Riordan, 2009)
, where they identify a threshold probability forpercolation of cliques in for all fixed and , which is given by . Moreover, they proved that for around this threshold, the number of cliques asymptotically converge to a Poisson distribution. Exceeding this threshold results in formation of giant connected clique clusters, which causes an explosion in the number of possible cliques.
Recall from our definition of , that two cliques are adjacent if they share at least vertices. In order to analyze this further, we imagine an entry in occurs when we can migrate a -clique from its original position to an adjacent clique by relocating exactly vertices and leaving the remaining vertices intact. The expected number of such relocations is given by , where the first term denotes the number of possible vertices in a -clique that can be chosen for relocation, the second term counts the number of new adjacent positions a clique can relocate to, and the final term decides the probability of relocations that are correct and acceptable. In our case, we define cliques to be adjacent to one another when they share at least vertices. This is done in order to keep the number of adjacent cliques to a manageable size during experiments. Setting , gives expected relocations, which in turn estimates .
Note that for every iteration in Algorithm 1, the dominating cost is that of running the Kuhn-Munkres matching algorithm in Step , which has a cubic cost in . Let denote an upper bound on all the number of non-zero entries in . Then, every iteration has a runtime and therefore after iterations the final cost is . Observe that as the dimensionality of the cliques increases in every iteration, decays very sharply and hence drastically reduces , which in turn reduces the overall matching cost. Finally, the storage complexity can simply be given as .
3 Theoretical Analysis of QAP
In this section, we present three related results in the context of matching random matrices, namely: (i) concentration inequality of eigenvalue bounds on the QAP trace formulation for random symmetric matrices, (ii) tighter concentration inequality of eigenvalue bounds on the Lawler QAP formulation on random symmetric matrices in the context of works that use affinity matrices, and (iii) provide an asymptotic analysis on the worst to best case ratio of a QAP for higher-dimensional clique adjacency matrices. For ease of notation, we will refer to the random clique adjacency matrices simply as and .
3.1 Eigenvalue Bounds of Trace QAP Formulation on Random Matrices
Let , be random real-symmetric matrices. Let be a permutation matrix. Then, the trace formulation of a QAP is given by
where is the set of permutation matrices.
Let and 111The two sets of eigenvalues differ in ordering. be the eigenvalues of and , respectively. Let the corresponding eigen-decompositions of matrices and , be given by and , where and
with their corresponding orthogonal eigenvector matricesand . Finke et. al. (Martello et al., 1987) gave the following eigenvalue bounds.
Let and be symmetric matrices. Then for all ,
where with vectors of eigenvalues given by and . and denote the -th eigenvectors of and , respectively;
It was also noticed by Finke (Martello et al., 1987) that these bounds can further be tightened by reducing the spreads of matrices and , where the spread of a matrix , denoted by , is given by . There is no formula to compute the spread of a matrix directly, so Finke et. al. (Martello et al., 1987) suggested a reduction method to further sharpen the bound by replacing matrices and by smaller spread symmetric matrices and . The reductions are achieved as and , where , are matrices with constant columns and , are diagonal matrices, whose values are chosen appropriately in order to tighten the bounds on spreads and .
Our bounds on Random Matrices
: We propose new measure concentration inequalities on the spread of a random matrix, by redefining the spread in an alternate fashion that is more amenable to our analysis. Consider our reduced random symmetric matrix, with eigenvalues , we define the gap (spacing) between its consecutive eigenvalues as for . Then, the spread for the reduced matrix can be redefined as:
We begin by upper bounding using the following lemma 1 (proof in supplementary notes).
Let denote an algebraic matrix norm on a space of real matrices , then for any ,
To the best of the author’s knowledge there does not exist a known distribution of eigenvalue gaps for a symmetric random matrix. We now attempt to give concentration inequalities for the tail probabilities of the sum of eigenvalue gaps, i.e., the spread. For our i.i.d. random matrix , consider the sequence of independent eigenvalue gaps , where each is upper bounded by , as shown in Lemma 1. Let us denote their sum as . As are independent scalar random variables with a.s, with mean
and variance. Then, using Chernoff bounds, for any , we have
for some absolute constants . The Chernoff inequality above, shows that is sharply concentrated in the range , when is not too large.
3.2 Eigenvalue bounds on Lawler’s QAP on Random Affinity Matrices
In literature, many graph matching algorithms use Lawler’s QAP formulation. Recall, . Let denote the pairwise affinity score of assigning the -th entry in to the -th entry in , implying that node is matched to node and node to node , simultaneously. Then, the affinity matrix is given by and the optimal assignment to Lawler’s QAP is the one that maximizes the sum total pairwise affinity scores. Leordeanu et. al. (Leordeanu & Hebert, 2005) show via a spectral relaxation that Lawler’s WAP reduces to solving , . This is solved by finding the leading eigenvalue .
For a random affinity matrix and for a positive constant , , where is the median of .∎
Discussion: We further investigate the robustness of affinity-matrix based graph matching solutions when dealing with missing or incomplete data. We show the sharpness of our result in Theorem 2 on the affinity matrix, similar to (Alon et al., 2002), who analyze their results using fat matrices as an example. Consider an affinity matrix
, whose entries are i.i.d. Bernoulli distributed. Simulating missing affinity scores due to missing edge assignments, we setwith probability and with probability . Notice that our random affinity matrix represents the Erdős-Rényi graph . As shown in (Alon et al., 2002), the median and expectation of differ by a constant factor. Let denote a general undirected graph, where the degree of each vertex is given by . Then, the average degree of is given by and its maximum degree is . It is then well known that , i.e., the largest eigenvalue of a graph is squeezed between its average and maximum degree.
Let denote the total number of edges in , then the average degree of is given by , where
and the standard deviation ofis . For large to deviate from its expectation by standard deviations as . Furthermore, we know that if exceeds its expectation by , then the average degree must also correspondingly exceed its expectation by . Therefore, the probability of the average degree exceeding its expectation by standard deviations is at least . Given that , it follows that exceeding its expectation is also lower bounded by the same . The bounds achieved are tight up to a constant factor in the exponent. Our experimental results on Factorized Graph Matching (FGM) by Zhou et. al. (Zhou & De la Torre, 2016) and Re-weighted Random Walk Matching (RRWM) (Cho et al., 2010) also support the finding that affinity matrix based matching solutions are more robust to missing edges due to occlusions in data.
3.3 Asymptotic Analysis of Higher-Order Clique Assignment
Following along the same lines as Finke et. al. (Martello et al., 1987), we study the asymptotic behavior of the worst to most optimal ratio and present it as the following theorem.
Given random clique adjacency matrices and their associated cost matrix . We denote by and
the expectation and variance of our Poisson distributed cost function.
the expectation and variance of our Poisson distributed cost function. Forand , we have the following bound on the ratio of the worse to the best solution as
where, , , ∎
|OurMethod||0.01 0.0||0.01 0.0||0.05 0.0||0.03 0.0||0.0 0.0||0.0 0.0||0.1 0.0||0.2 0.1||0.0 0.0||0.0 0.0|
|EigenAlign||63.97 1.0||70.19 1.0||65.39 1.5||70.88 1.9||62.5 0.4||64.97 0.2||62.37 0.6||65.42 1.1||61.04 0.2||62.36 0.7|
|FGM||3.6 0.5||7.4 0.5||18.0 0.0||36.4 0.5||0.0 0.0||0.0 0.0||2.20 1.0||3.4 0.5||0.0 0.0||0.0 0.0|
|LAI-LP||40.90 0.9||43.07 1.2||49.9 1.2||61.16 0.7||49.07 0.9||59.41 0.8||43.12 5.5||45.91 3.2||39.04 0.7||38.27 0.6|
|PermSync||13.36 1.1||16.48 0.4||25.48 1.4||41.59 0.7||26.24 3.4||42.31 5.1||12.15 0.7||13.7 0.6||10.19 0.8||6.23 3.2|
|RRWM||2.0 0.0||4.0 0.0||14.0 0.0||27.0 0.0||0.0 0.0||0.0 0.0||5.0 2.0||10.0 1.5||0.0 0.0||0.0 0.0|
|Tensor||6.88 0.0||14.39 0.8||19.24 0.6||37.95 0.8||17.1 0.5||34.69 0.4||2.91 0.6||6.19 0.7||0.0 0.0||0.0 0.0|
|IPFP||6.8 1.5||11.6 0.5||18.2 0.5||35.0 0.0||1.0 0.0||1.0 0.0||7.4 2.0||15.6 1.5||0.8 0.5||0.6 0.5|
|PM||43.2 1.0||49.4 1.5||46.4 1.0||56.8 0.5||37.0 0.0||37.0 0.0||45.6 0.5||53.4 0.5||37.8 0.5||38.4 0.5|
|SMAC||12.6 0.5||20.0 0.0||19.0 0.0||33.0 0.0||4.0 0.0||4.0 0.0||13.8 2.5||23.4 0.5||4.0 0.0||4.2 0.5|
|SM||32.6 0.5||39.4 1.5||37.8 1.0||49.6 1.0||25.0 0.0||25.0 0.0||33.0 1.5||40.4 0.5||25.2 0.5||24.8 0.5|
|GA||34.0 1.5||37.0 1.0||38.4 1.0||47.0 0.0||30.0 0.0||30.0 0.0||37.6 1.0||45.4 0.5||29.0 0.0||28.4 0.5|
|Munkres||35.25 1.5||36.47 0.9||44.75 0.8||55.13 1.3||46.05 1.2||57.25 2.2||37.32 1.5||37.81 1.8||32.84 0.3||31.27 0.8|
Here, we study the robustness of various matching algorithms when affected by missing or incomplete information and transformations (both affine and non-affine) on synthetic and real-world datasets. For the sake of brevity, we report detailed dataset descriptions in our supplementary notes. The graph matching algorithms can broadly be classified based on their use of (i) affinity-matrix: FGM (Zhou & De la Torre, 2016; Zhou & De la Torre, 2013)222FGM, RRWM (Cho et al., 2010), (ii) Eigenvalues: EigenAlign (Feizi et al., 2016)333EigenAlign, SM (Leordeanu & Hebert, 2005), SMAC (Cour et al., 2007), PermSync (Pachauri et al., 2013)444PermSync, (iii) LP relaxation: GA (Gold & Rangarajan, 1996), Kuhn-Munkres (Kuhn, 1955), (iv) Integer QAP: IPFP (Leordeanu et al., 2009), (v) Probabilistic matching: PM (Zass & Shashua, 2008), (vi) Higher-order matching given complete data: Tensor (Duchenne et al., 2011)555Tensor, and (vii) Geometric and Feature matching: LAI-LP (Li et al., 2013)666This algorithm serves as our naive baseline as it directly uses neighborhood properties of the underlying graph (LAI-LP)..
4.1 Effect of Affine Transformations
Simulated Dataset: We perform affine transformations on CMU House, which is a sequence of frames extracted from a video. More specifically, we uniformly sample frames (at and ) and perform affine transformations on the selected frames to distort them. Figure 2 shows examples of affine transformations on house frame sequences. Table 1 shows the comparative error in matching for all the algorithms. We now describe each affine transformation as performance metrics in our experiments.
|OurMethod||4.14 2.45/ 7.13||3.15 0.32/ 6.96||3.89 0.23/ 14.76||0.48 0.02/ 43.99||4.17 0.32/ 12.65||22.20 1.16/ 14.86|
|EigenAlign||60.68 0.29/ 19.37||57.44 0.37/ 19.32||66.57 0.0/ 26.13||43.23 0.0/ 93.60||90.51 0.0/ 2.64||98.41 0.0/ 8.29|
|FGM||55.51 0.0/ 1793.9||48.17 0.0/ 2013.7||16.12 0.0/ 674.94||0.0 0.0/ 777.55||74.87 0.05/ 2530.5||97.54 0.01/ 4293.9|
|LAI-LP||73.06 0.23/ 152.47||42.00 0.24/ 154.15||49.54 0.0/ 161.06||88.73 0.1/ 184.153||87.98 0.0/ 33.06||96.38 0.0/ 14.71|
|PermSync||10.63 0.0/ 0.45||8.90 0.0/ 0.46||46.93 0.0/ 0.43||79.88 0.0/ 1.08||64.00 0.0/ 0.22||70.00 0.0/ 0.48|
|RRWM||60.91 0.0/ 4.96||54.53 0.0/ 4.83||30.99 0.0/ 8.53||1.98 0.0/ 18.09||72.87 0.01/ 7.98||87.04 0.0/ 21.84|
|Tensor||24.37 0.9/ 93.36||15.07 1.0/ 93.97||1.07 0.17/ 107.93||0.0 0.0/ 182.07||43.24 2.98/ 40.21||32.35 0.15/ 40.41|
|IPFP||65.13 0.0/ 6.35||60.81 0.0/ 6.28||40.90 0.0/ 8.43||3.94 0.0/ 12.31||76.19 0.0/ 4.65||87.74 0.0/ 8.90|
|PM||74.63 0.0/ 7.07||71.93 0.0/ 4.90||70.27 0.0/ 0.94||48.82 0.0/ 1.69||83.79 0.02/ 2.98||91.43 0.24/ 0.44|
|SMAC||70.00 0.0/ 5.75||67.36 0.0/ 5.52||50.53 0.0/ 4.15||5.52 0.0/ 6.90||78.56 0.22/ 1.94||87.88 0.11/ 3.42|
|SM||68.54 0.0/ 3.34||67.18 0.0/ 3.47||65.96 0.0/ 3.32||34.16 0.0/ 4.93||80.27 0.07/ 1.88||88.66 0.09/ 2.10|
|GA||65.06 0.0/ 4.53||64.60 0.0/ 4.61||61.58 0.0/ 4.08||31.62 0.0/ 5.83||77.20 0.27/ 3.51||87.02 0.16/ 32.28|
|Munkres||33.71 0.0/ 1.52||29.99 0.0/ 1.49||51.87 0.0/ 1.39||79.69 0.0/ 2.45||74.00 0.0/ 0.73||92.00 0.0/ 1.16|
Rotation: Figure 2 shows a rotated version of the original house frame (Figure 2). Table 1 shows errors in matching when % of the frames are rotated by both and , respectively and when the same transformations are applied to % of the frames. As the percentage of transformed frames with greater degree increases, we note a substantial increase in error for other methods in comparison to our method’s error increase.
Scaling: Resizing an image both horizontally and vertically scales the image as is shown in Figure 2 . We fixed the scales to , , , and randomly in both the directions in order to transform the images. Our method in Table 1 produces much better matchings than the other methods.
4.2 Effect of Incomplete and Occluded Landmarks
To understand the effect of occlusions, we took two real-world datasets, i.e., Books and Building (Pachauri et al., 2013) with severe occlusions which are scenes of the same 3D object taken from arbitrary camera angles. These datasets have widely been used in Structure from Motion (SfM) problems and are known to be difficult for matching. Focusing our attention to the last two columns of Table 2, it is evident that our method gives the best results.
Figure 3 shows the Books dataset where books are placed on a table in various orientations with varying levels of occlusion, along with two sample matchings between different pairs of images. Note that in Figure 3, when a corresponding matching clique is not found in the other image, a match isn’t forced but rather there is no match reported, which doesn’t degrade the matching accuracy. Matching as many random cliques, in order of decreasing dimensionality, as possible, manifests itself as an advantage over existing methods, especially when dealing with clutter and/or occlusions.
Simulating missing points: In order to gain a deeper insight into the behavior of all the matching algorithms, we omit , and (, , , , and ) points out of total House landmark points (i.e., points) from (Figure 4) of frame sequences randomly. In general, all algorithms show an increase in error as more points are removed, but our method has a less gradual increase, while eigenvalue related methods show a rather steep increase in error. Our method is comparable to FGM and RRWM, but the gap in error increases with more missing points. We also observed that FGM incurs the longest runtime for matching in this scenario.
4.3 Effect of Frame Separation
Here, we pick two frames from a video for matching and vary the separation in their frame sequence number. The farther apart two frames are the more pronounced is the effect we seek between the frame images. For example, as the frame separation increases, CMU Hotel undergoes a more severe 3D rotation, while Horse-Shear (Caetano et al., 2009) undergoes a larger degree of shear.
We set and as nearest neighbors to get the correct matchings. In Figure 5, in both the left and right plots we notice that most methods show a very sharp rise in error, while our method is quite stable and reports a error. We observe that the naive baseline, LAI-LP also does well and doesn’t exhibit steep changes in error with larger frame separation.
Experimental Summary: In general, we find that the affinity matrix based methods like FGM and RRWM are more robust to affine transformations than other competing algorithms. Our method performs the best as the weight vectors in our algorithm effectively capture even the higher-order geometric properties of the neighborhood and nearly preserves them under affine transformations. The naive baseline, i.e., LAI-LP, does not perform as well because it also has a feature-based component like SIFT which is known to fail on some affine transformations.
To the best of our knowledge, we have presented the first approach towards partial higher-order matching by initially capturing higher-order structure as random clique complexes and then proposing a corresponding matching algorithm. From a theoretical point of view, we studied matching as a QAP on random clique adjacency matrices that represented the -skeleton of our random clique complexes and gave bounds on the concentration inequality of the spread of its eigenvalues. We also improved bounds on the largest eigenvalue of the Lawler QAP formulation, used by affinity-matrix based approaches. We discussed the robustness of such approaches to missing points and also showed the sharpness of our result. Furthermore, inspired by Finke et. al. (Martello et al., 1987) we studied the asymptotic behavior of our higher dimensional clique adjacency matrices. A more detailed investigation of the distribution of eigenvalue gaps for such random matrices with Poisson distributed entries is left for future work.
From an empirical perspective, we compared the matching accuracies of diverse algorithms on both synthetic and real-world datasets that were known to have severe occlusions and distortions, thus posing a daunting challenge to matching algorithms. We argue that our experiments show strong evidence that our approach outperforms all the state-of-the-art matching methods on a diverse range of datasets.
We thank our colleagues from the Mathematics Dept. at IIT-H (Sukumar Daniel, Narasimha Kumar, and Bhakti B. Manna) for their insight and expertise. We would also like to thank all the reviewers for their feedback and suggestions. We are grateful to the authors of (Zhou & De la Torre, 2016; Zhou & De la Torre, 2013; Cho et al., 2010; Feizi et al., 2016; Pachauri et al., 2013; Li et al., 2013; Duchenne et al., 2011) for providing their source codes and datasets.
- Alon et al. (2002) Alon, N., Krivelevich, M., and Vu, V. H. On the concentration of eigenvalues of random symmetric matrices. Israel Journal of Mathematics, 131(1):259–267, Dec 2002.
Berg et al. (2005)
Berg, A. C., Berg, T. L., and Malik, J.
Shape matching and object recognition using low distortion
Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pp. 26–33. IEEE, 2005.
- Bollobás & Riordan (2009) Bollobás, B. and Riordan, O. Clique percolation. Random Struct. Algorithms, 35(3):294–322, 2009.
- Caetano et al. (2009) Caetano, T. S., McAuley, J. J., Cheng, L., Le, Q. V., and Smola, A. J. Learning graph matching. IEEE transactions on pattern analysis and machine intelligence, 31(6):1048–1058, 2009.
- Chertok & Keller (2010) Chertok, M. and Keller, Y. Efficient high order matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(12):2205–2215, Dec 2010.
- Cho et al. (2010) Cho, M., Lee, J., and Lee, K. M. Reweighted random walks for graph matching. In European conference on Computer vision, pp. 492–505. Springer, 2010.
- Cho et al. (2013) Cho, M., Alahari, K., and Ponce, J. Learning graphs to match. In Proceedings of the IEEE International Conference on Computer Vision, pp. 25–32, 2013.
Conte et al. (2004)
Conte, D., Foggia, P., Sansone, C., and Vento, M.
Thirty years of graph matching in pattern recognition.
International journal of pattern recognition and artificial intelligence, 18(03):265–298, 2004.
- Cour et al. (2007) Cour, T., Srinivasan, P., and Shi, J. Balanced graph matching. In Advances in Neural Information Processing Systems, pp. 313–320, 2007.
- Duchenne et al. (2011) Duchenne, O., Bach, F., Kweon, I.-S., and Ponce, J. A tensor-based algorithm for high-order graph matching. IEEE transactions on pattern analysis and machine intelligence, 33(12):2383–2395, 2011.
- Feizi et al. (2016) Feizi, S., Quon, G., Recamonde-Mendoza, M., Médard, M., Kellis, M., and Jadbabaie, A. Spectral alignment of networks. arXiv preprint arXiv:1602.04181, 2016.
- Giusti et al. (2015) Giusti, C., Pastalkova, E., Curto, C., and Itskov, V. Clique topology reveals intrinsic geometric structure in neural correlations. Proc. Nat. Acad. Sci. USA, 112(44):13455–13460, 2015.
- Goesele et al. (2007) Goesele, M., Snavely, N., Curless, B., Hoppe, H., and Seitz, S. M. Multi-view stereo for community photo collections. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pp. 1–8. IEEE, 2007.
- Gold & Rangarajan (1996) Gold, S. and Rangarajan, A. A graduated assignment algorithm for graph matching. IEEE Transactions on pattern analysis and machine intelligence, 18(4):377–388, 1996.
- Jiang et al. (2011) Jiang, H., Stella, X. Y., and Martin, D. R. Linear scale and rotation invariant matching. IEEE transactions on pattern analysis and machine intelligence, 33(7):1339–1355, 2011.
- Kahle (2006) Kahle, M. Topology of random clique complexes, 2006. URL http://arxiv.org/abs/math.CO/0605536.
- Kuhn (1955) Kuhn, H. W. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83–97, 1955.
- Leordeanu & Hebert (2005) Leordeanu, M. and Hebert, M. A spectral technique for correspondence problems using pairwise constraints. In Proceedings of the Tenth IEEE International Conference on Computer Vision - Volume 2, ICCV ’05, pp. 1482–1489, 2005.
- Leordeanu et al. (2009) Leordeanu, M., Hebert, M., and Sukthankar, R. An integer projected fixed point method for graph matching and map inference. In Advances in neural information processing systems, pp. 1114–1122, 2009.
- Li et al. (2013) Li, H., Huang, X., and He, L. Object matching using a locally affine invariant and linear programming techniques. IEEE transactions on pattern analysis and machine intelligence, 35(2):411–424, 2013.
- Martello et al. (1987) Martello, S., Minoux, M., Ribeiro, C., and Laporte, G. Surveys in Combinatorial Optimization, Volume 31. jan 1987.
- Pachauri et al. (2013) Pachauri, D., Kondor, R., and Singh, V. Solving the multi-way matching problem by permutation synchronization. In Advances in neural information processing systems, pp. 1860–1868, 2013.
- Petterson et al. (2009) Petterson, J., Yu, J., McAuley, J. J., and Caetano, T. S. Exponential family graph matching and ranking. In Advances in Neural Information Processing Systems, pp. 1455–1463, 2009.
- Shen & Davatzikos (2002) Shen, D. and Davatzikos, C. Hammer: hierarchical attribute matching mechanism for elastic registration. IEEE transactions on medical imaging, 21(11):1421–1439, 2002.
- Singer & Shkolnisky (2011) Singer, A. and Shkolnisky, Y. Three-dimensional structure determination from common lines in cryo-em by eigenvectors and semidefinite programming. SIAM journal on imaging sciences, 4(2):543–572, 2011.
- Szeliski (2010) Szeliski, R. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
- Talagrand (1995) Talagrand, M. Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathématiques de l’Institut des Hautes Études Scientifiques, 81(1):73–205, 1995.
- Zaslavskiy et al. (2009) Zaslavskiy, M., Bach, F., and Vert, J.-P. Global alignment of protein–protein interaction networks by graph matching methods. Bioinformatics, 25(12):i259–1267, 2009.
- Zass & Shashua (2008) Zass, R. and Shashua, A. Probabilistic graph and hypergraph matching. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8. IEEE, 2008.
- Zhou & De la Torre (2013) Zhou, F. and De la Torre, F. Deformable graph matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2922–2929, 2013.
- Zhou & De la Torre (2016) Zhou, F. and De la Torre, F. Factorized graph matching. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 38(9):1774–1789, 2016.