A Genetic Algorithm for Fully Automatic Non-Isometric Shape Matching

05/26/2019 ∙ by Michal Edelstein, et al. ∙ 0

Automatically computing shape correspondence is a difficult problem, especially when the shapes are significantly different. In this paper we suggest a fully automatic method for shape correspondence, that is suitable for non isometric shapes and shapes of different topology. We tackle the combinatorial task of putting in correspondence two sparse sets of landmarks using a genetic algorithm. Our main observation is that optimizing an objective based on an induced dense functional correspondence, combined with geometric genetic operators, is highly effective for non isometric shape matching. The output of the genetic algorithm is a sparse landmark correspondence, as well as a corresponding functional map. Finally, an accurate pointwise map is extracted using existing semi-automatic methods. Our method is general, widely applicable, and outperforms state of the art methods for automatic shape correspondence both quantitatively and qualitatively.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 3

page 4

page 5

page 6

page 7

page 8

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Shape correspondence, or shape matching, is a fundamental task in shape analysis. Given two shapes, the goal is to compute a semantic correspondence between points on them. Shape correspondence is required when two shapes are analyzed jointly, which is common in many applications such as texture and deformation transfer [SP04], statistical shape analysis [MDW08] and shape classification [ESKBC17], to mention just a few examples. There are different categories of shape matching methods: correspondence between rigid shapes, non rigid isometric correspondence, which characterizes shapes of the same articulated object in different poses, and general correspondence between shapes that belong to the same semantic class but are not necessarily isometric. In this paper we devise a fully automatic method for the latter, i.e., for non isometric shape matching between manifold triangle meshes.

It is instrumental to consider the non-isometric shape correspondence problem as a two step process. First, the global semantics of the matching are given by a sparse set of corresponding landmarks, which match salient points on both shapes. If this set is informative enough, then the full shapes can be matched by extending the landmark correspondence to a full map from the source to the target in a consistent and smooth way. The first problem is combinatorial, requiring the computation of a permutation of a subset of the landmarks, whereas the second problem is continuous, requiring the definition and computation of local differential properties of the map. Whereas the second problem has been tackled by multiple methods [AL16, MCSK17, ESBC19, EHA19] which yield excellent results for non-isometric shapes, methods that address the sparse landmark correspondence problem [KKBL15, MDK16, DML17, Sah18] have so far been limited either to the nearly isometric case, or to a very small set of landmarks.

Our main contribution is a novel approach to computing a sparse landmark correspondence, and its extension to a full dense map, using a genetic algorithm

. Genetic algorithms have been used for combinatorial optimization for a few decades 

[Hol92], and are quite general in the type of objectives they can optimize. However, to the best of our knowledge, their use in shape analysis has been limited so far to isometric matching [Sah18]. In the context of our problem, using a genetic algorithm allows us to optimize a complex non-convex objective, that is given both in terms of the landmark permutation and the differential properties of the extended map computed from these landmarks. We use a non-linear non-convex objective, given by the elastic energy of the deformation of the shapes, an approach that has been recently used successfully for non-isometric matching [EHA19] when the landmarks are known. We additionally design operators for the genetic algorithm which are geometric in nature, and allow to merge and mutate given sparse correspondences. We show that our algorithm converges to a landmark correspondence that can be extended to a functional map and a full vertex-to-point map, which outperforms existing state-of-the-art techniques for automatic shape correspondence both quantitatively and qualitatively.

1.1 Related Work

As the literature on shape correspondence is vast, we discuss here only methods which are directly relevant to our approach. For a more detailed survey on shape correspondence we refer the reader to the recent excellent reviews [VKZHCO11, TCL13].

Fully automatic shape correspondence.

Many fully automatic methods, like ours, compute a sparse

correspondence between the shapes, to decrease the number of degrees of freedom and possible solutions. A sparse-to-dense method can be then used in a post processing step to obtain a dense map. For example, Kezurer et al. kezurer2015tight formulated the shape correspondence problem as a Quadratic Assignment Matching problem, and suggested a convex semi-definite programming (SDP) relaxation to solve it efficiently. While the convex relaxation was essential, their method is only suitable for small sets (of the same size) of corresponding landmarks. Dym et al. DSppDym suggested to combine doubly stochastic and spectral relaxations to optimize the Quadratic Assignment Matching problem, which is not as tight as the SDP relaxation, but much more efficient. Maron et al. Maron2016point suggested a convex relaxation to optimize a term that relates pointwise and functional maps, which promotes isometries by constraining the functional map to be orthogonal.

Other methods for the automatic computation of a dense map include Blended Intrinsic Maps (BIM) by Kim et al. [KLF11], who used the most isometric combination of conformal maps. Their method works well for relatively similar shapes and generates locally smooth results, yet is restricted to mapping between genus- surfaces. Vestner et al. [VLR17] suggested a multi scale approach that is not restricted to isometries, but requires shapes with the same number of vertices and generates bijective vertex to vertex correspondence.

A different approach to tackle the correspondence problem is to compute a fuzzy map [OBCS12, SPKS16]

. The first approach puts in correspondence functions instead of points, whereas the second is applied to probability functions. These generalizations allow much more general types of correspondences, e.g. between shapes of different genus, however, they also require an additional pointwise map extraction step.

The functional map approach was used and extended by many following methods, for example Nogneng et al. [NO17] introduced a pointwise multiplication preservation term, Huang et al. [HO17] used the adjoint operators of functional maps, and Ren et al. [RPWO18] recently suggested to incorporate an orientation preserving term and a pointwise extraction method that promotes bijectivity and continuity, that can be used for non isometric matching as well.

Our method differs from most existing methods by the quality measure that we optimize. Specifically, we optimize for the landmark correspondence by measuring the elastic energy of the dense correspondence implied by these landmarks. As we show in the results section, our approach outperforms existing automatic state-of-the-art techniques on challenging datasets.

Semi-automatic shape correspondence.

Many shape correspondence methods use a non-trivial initialization, e.g. a sparse landmark correspondence, to warm start the optimization of a dense correspondence. Panozzo et al. [PBDSH13] extended the landmark correspondence by computing the surface barycentric coordinates of each point w.r.t. the landmarks on the source shape, and matching it to the point on the target shape with the same barycentric coordinates w.r.t. the target landmarks. The landmarks could be chosen interactively, which allowed for intuitive user control. More recently, Gehre et al. [GBKS18], used curve constraints and functional maps, for computing correspondences in an interactive setting. Given landmark correspondences or an extrinsic alignment, Mandad et al. [MCSK17]

used a soft correspondence, where each source point is matched to a point on the target shape with a certain probability, and minimized the variance of this distribution. Parameterization based methods map the two shapes to a single, simple domain, such that the input landmark correspondence is preserved, and the composition of the maps from each shape to the common domain gives the final result 

[APL15, AL15, AL16]. Since minimizing the distortion of the map to the common domain does not guarantee minimal distortion of the final map, recently Ezuz et al. [ESBC19] directly optimized the Dirichlet energy and the reversibility of the forward and backward maps, which led to results with low conformal distortion. Finally, many automatic methods, for example all the functional map based approaches, can use landmarks as an auxiliary input to improve results in highly non isometric settings.

The output of our method is a sparse correspondence and a functional map, that can be used as input to semi-automatic methods to generate a dense map, thus our approach is complementary to semi-automatic methods. Specifically, we use [ESBC19] to extract a dense vertex-to-point map from the landmarks and functional map we compute with the genetic algorithm.

Genetic algorithms.

Genetic algorithms were initially inspired by the process of evolution and natural selection [Hol92]. In the last few decades they were used in many areas, such as protein folding simulations [UM93], clustering [MB00] and image segmentation [BLM95]. In the context of graph and shape matching, genetic algorithms were used for registration of depth images [CTL04, SBB05], 2D shape recognition in images [OM97], rigid registration of 3-D curves and surfaces [YAF99], and inexact graph matching [CWH97, Auw07].

More recently, Sahillioğlu SahilliogluGenetic suggested a genetic algorithm for isometric shape matching. However, their approach is drastically different from ours. Their objective was the preservation of pairwise distances between the sparse landmarks. We, on the the other hand, use as our objective the energy of a dense correspondence, which is the output of a sparse-to-dense algorithm. The different focus of our approach is also evident in the other genetic operators that we define, which allow us to compute a highly non isometric matching.

1.2 Overview and Contribution

Our general pipeline is described in Section 3, and the following sections provide more details. We start by selecting geometrically meaningful landmarks on each shape (Section 4). Since we do not expect the landmark sets on the two shapes to match exactly, in the next step we compute a partial landmark correspondence, where each source landmark can be either matched to a unique target landmark or remained unmatched. We compute the landmark correspondence, as well as a functional map, using a genetic algorithm, as described in Section 5. Finally, we use a post processing step that extends the solution to a dense smooth correspondence. In Section 6 we demonstrate our results. We compare with other state of the art methods for automatic sparse correspondence [KKBL15, DML17, Sah18], and with the recent functional map based method [RPWO18], that automatically computes dense maps and does not have topological restrictions. We apply the same post-processing for computing a sparse-to-dense map to all methods, and show that we outperform previous methods, as demonstrated by both quantitative and qualitative evaluation.

2 Background

2.1 Notation

We use the following notation throughout the paper. We compute a correspondence between two manifold triangle meshes, denoted by and , with vertices respectively. The vertex, edge and face sets are denoted by respectively, where the subscript indicates the corresponding mesh. The embedding in of the mesh is given by . The area of a face is denoted by , and similarly is the area of a vertex (defined to be a third of the area of its adjacent faces).

A map that assigns a point on to each vertex of is denoted by , the corresponding matrix is (see section 2.2). Similarly, maps in the opposite direction are with opposite subscript, e.g. is a pointwise map from to .

The eigenfunctions of the Laplace-Beltrami operator that correspond to the smallest

eigenvalues are used as reduced bases for scalar functions, and they are stacked as the columns of the matrices . For a vertex , its corresponding row in the basis matrix is denoted by . The functional map in this reduced basis is denoted by (see equation (1)).

2.2 Functional Maps

Formulation.

The term functional maps, as first introduced by Ovsjanikov et al. [OBCS12, OCB16], refers to a linear operator that maps scalar functions on one shape to the other. A pointwise map from to can be fully represented by a matrix , where at each row there are at most 3 non zero entries, at entries that correspond to vertices of the face on that is mapped to, and the values are the barycentric coordinates. A piecewise linear function on

, represented by a vector of vertex values

, can be transported to by . It assigns to each vertex of the value of at the point it is mapped to, . By projecting on reduced bases for scalar functions, of size , of size respectively, we get a compact functional map:

(1)

Similarly, given a functional map , one option to convert it to a pointwise map is to use:

(2)

Note, that there are multiple ways to extract a pointwise map from a functional map, see e.g. the discussion in [EBC17].

Basis.

A common choice of basis for these subspaces is given by the first eigenfunctions of the Laplace-Beltrami (LB) operator of each shape, such that smooth functions can be well approximated using a small number of coefficients and is compact.

It is often valuable to use a larger basis size for the target functions, so that mapped functions are well represented. Hence, since maps functions on to functions on , should contain more basis functions than . Thus, we denote the number of source eigenfunctions by , the number of target eigenfunctions by , and the functional maps in both directions are of size . Slightly abusing notations, and to avoid clutter, we use to denote the eigenfunctions corresponding to , in both directions, namely both when and , as the meaning is often clear from the context. Where required, we will explicitly denote by the eigenfunctions with dimensions , respectively. In this paper we always take .

Objectives.

Many cost functions have been suggested for functional map computation, e.g., [OBCS12, NO17, CMS18, RPWO18], among others. In our approach we use the following terms.

Landmark correspondence.

Given a set of pairs of corresponding landmarks, , we use the term:

(3)

While some methods use landmark-based descriptors, we prefer to avoid it due to possible bias towards isometry that might be inherent in the descriptors. The formulation in Equation (3) has been used successfully by Gehre et al. [GBKS18] for functional map computation between highly non isometric shapes, as well as in the context of pointwise map recovery [EBC17].

Commutativity with Laplace-Beltrami.

We use:

(4)

where is a diagonal matrix with the first eigenvalues of the Laplace-Beltrami operator of on the diagonal. While initially this term was derived to promote isometries, in practice it has proven to be useful for highly non isometric shape matching as well [GBKS18].

2.3 Elastic Energy

Elastic energies are commonly used for shape deformation [BPGK06, SA07, HRWW12, HRS14], but were used for shape matching as well [LDRS05, WSSC11, dBDFN16, IRS18, EHA19]. In this paper we use a recent formulation that achieved state of the art results for non isometric shape matching [EHA19]. The full details are described there, and we mention here only the main equations for completeness.

The elastic energy is defined for a source, or undeformed mesh , and a deformed mesh with the same triangulation but different geometry . It consists of two terms, a membrane term and a bending term. The membrane energy penalizes area distortion:

(5)

where denotes the area of face , and

denotes the geometric distortion tensor of the face

. In addition, in order to have a finite expression in case some triangles are deformed into zero area triangles, the negative function is linearly extended below a small threshold.

The bending energy, on the other hand, penalizes misalignment of curvature features:

(6)

where denote the dihedral angle at edge in the undeformed and deformed surfaces respectively; if are the adjacent triangles to then , and is the length of in the deformed surface.

The total elastic energy is:

(7)

where we always use .

In the context of shape matching evaluation, the undeformed mesh is , and the geometry of the deformed mesh is given by the embedding of the points on that correspond to the vertices of . Specifically, these are given by , or equivalently, , where is the embedding of the vertices of .

2.4 Energies of Functional Maps

Elastic energy.

The elastic energy can also be used to evaluate functional maps directly, by setting the geometry of the deformed mesh to . For brevity, we denote this energy by .

Reversibility energy.

In [EHA19] the elastic energy was symmetrized and combined with a reversibility term that evaluates bijectivity. The reversibility term requires computing both and . Here we define the reversibility energy for functional maps, similarly to [ERGB16]:

(8)

The reversibility energy measures the distance between vertex coordinates (projected on the reduced basis), and their mapping to the other shape and back. The smaller this distance is, the more bijectivity is promoted [ESBC19].

Figure 1: Our pipeline consists of three stages: (a) landmark computation (Section 4), (b) genetic algorithm (Section 5), (c) sparse to dense post processing using [ESBC19].

3 Method

Our goal is to automatically compute a semantic correspondence between two shapes, denoted by and . The shapes are the only input to our method. We do not assume that the input shapes are isometric, but we do assume that both shapes belong to the same semantic class, so that a semantic correspondence exists.

Our pipeline consists of three main steps, see Figure 1:

  1. Compute two sets of geometrically meaningful landmarks on , denoted by (Section 4).

  2. Compute a sparse correspondence, i.e. a permutation, between , as well as corresponding functional maps , using a genetic algorithm (Section 5).

  3. Generate a dense pointwise map using an existing semi-automatic correspondence method [ESBC19].

We use standard techniques for the first and last steps, and thus our main technical contribution lies in the design of the genetic algorithm.

The most challenging component of our method is determining the objective function. Unlike the isometric correspondence case, where it is known that pairwise distances between landmarks should be preserved, in the general (not necessarily isometric) case there is no known criterion that the landmarks should satisfy. However, there exist well studied differential quality measures that proved to be useful in practice for dense non isometric correspondence. Therefore we use an existing method to extend the landmark correspondence to a dense correspondence, and define our objective function by the quality of the computed dense map.

To extend the landmark correspondence to a full dense correspondence, we chose a functional map approach, as it performs most computations in a reduced basis and is thus significantly faster than pointwise approaches. In our setting speed is of highest importance, as we compute a dense map for many different landmark correspondences.

The general formulation of the optimization problem we address is therefore:

(9)
subject to

where is a permutation, which maps a subset of the landmarks , to a subset of the landmarks , and is a functional map. Note that the objective that we measure for a given permutation , i.e. , is different than the objective that we optimize to extend to a full map, i.e., . Due to the genetic algorithm, which can be applied to very general objective functions, the first can be considerably more complicated, i.e. non-linear and non-convex, than the second.

In the following we discuss the details of the algorithm, first addressing the landmark computation, and then the design of the genetic algorithm that we use to optimize Equation (9).

4 Automatic Landmark Computation

This section describes the first step of our pipeline, automatic computation of geometrically meaningful landmarks on each shape. As a pre-processing step, we normalize both meshes to have area

. We classify the landmarks into three different categories, based on their computation method:

maxima, minima, and centers.

Maxima and Minima.

The first two categories are the local maxima and minima of the Average Geodesic Distance (AGD), that is frequently used in the context of landmark computation [KLF11, KKBL15]. The AGD of a vertex is defined as: where is the vertex area and is the geodesic distance between . To efficiently compute approximate geodesic distances, we compute a high dimensional embedding as suggested by Panozzo et al. [PBDSH13], and use the Euclidean distances in the embedding space that approximate geodesic distances. The maxima of AGD are typically located at tips of sharp features, and the minima at centers of smooth areas, thus the maxima of the AGD provide the most salient features.

Centers.

As the maxima and minima of the AGD are very sparse, we add additional landmarks using the local minima of the function:

(10)

defined by Cheng et al. [CMS18], who referred to these minima as centers. Here, and are the eigenfunction and eigenvalue of the Laplace-Beltrami operator, and is the number of eigenfunctions. We use the minima of Equation (10) rather than, e.g., farthest point sampling [KKBL15], as the centers tend to be more consistent between non isometric shapes.

Filtering.

Landmarks that are too close provide no additional information, and add unnecessary degrees of freedom. Hence, we filter the computed landmarks so that the minimal distance between the remaining ones is above a small threshold . When filtering, we prioritize the landmarks according their salience, namely we prefer maxima of the AGD, then minima and then centers. Finally, if the set of landmarks is larger than a maximal size of , we increase automatically to yield less landmarks. We take , and use as the initial threshold.

Adjacent landmarks.

We would like to define the genetic operators such that the geometry of the shapes is taken into consideration. Hence, we define two landmarks to be adjacent if their geodesic distance or if they are neighbors in the geodesic Voronoi diagram of . We always take . The set of adjacent landmarks to is denoted by .

Landmark origins.

The landmark categories are saved and used in the next step (the genetic algorithm) as well. Given a landmark , all the landmarks in which are from the same category are denoted by .

The maxima, minima and centers of each shape that remain after filtering form the landmark sets . The number of landmarks is denoted by , and is not necessarily the same for .

Figure 1(a) shows example landmark sets, where the color of a landmark indicates its type: blue for maxima, red for minima and green for centers. As expected, the landmarks do not entirely match, however there exists a substantial subset of landmarks that do match. At this point the correspondence between the landmarks is not known, and it is automatically computed in the next step, using the genetic algorithm.

5 Genetic Non Isometric Maps

Genetic algorithms are known to be effective for solving challenging combinatorial optimization problems with many local minima. In a genetic algorithm [Hol92] solutions are denoted by chromosomes, which are composed of genes. In the initialization step, a collection of chromosomes, known as the initial population is created. The algorithm modifies, or evolves, this population by selecting a random subset for modification, and then combining two chromosomes to generate a new one (crossover), and modifying (mutating) existing chromosomes. The most important part of a genetic algorithm is the objective that is optimized, or the fitness function. The ultimate goal of the genetic algorithm is to find a chromosome, i.e. a solution, with the best fitness value. The general genetic algorithm is described in Algorithm 1.

Genetic algorithms are quite general, as they allow the fitness function to be any type of function of the input chromosome. We leverage this generality to define a fitness function that is itself the result of an optimization problem. We further define the genes, chromosomes, crossover and mutation operators and initialization and selection strategies, in a geometric manner.

Figure 2: Illustration of chromosomes, which represent a sparse correspondence. The index of an array entry corresponds a landmark of (left), and the value corresponds to a landmark on (right shape). The first chromosome is the desired semantic correspondence, the second is valid but semantically incorrect, and the third is invalid because it is not injective (landmarks 2,3 on correspond to the same landmark with index 4 on ).

5.1 Genes and Chromosomes

Genes.

A gene is given by a pair such that and , and encodes a single landmark correspondence. If we denote it as an empty gene, otherwise it is a non-empty gene.

Adjacency Preserving Genes.

Two non-empty genes and are defined to be adjacency preserving (AP) genes, if are adjacent landmarks on , for .

Chromosome.

A chromosome is a collection of exactly genes, that includes a single gene for every landmark in . A chromosome is valid if it is injective, namely each landmark on is assigned to at most a single landmark in . We represent a chromosome using an integer array of size , and denote it by , thus the gene is encoded by (see Figure 2).

Match.

A match defined by a chromosome is denoted by and includes all the non-empty genes in . The sets are the landmarks that participate in the genes of , i.e., all the landmarks that have been assigned.

input : 
output : 
Initialize population ;
  // Section 5.2
Evaluate fitness ;
  // Section 5.3
while population did not converge do
       Select individuals for breeding ;
        // Section 5.4
       Perform crossover ;
        // Section 5.5
       Perform mutation ;
        // Section 5.6
       Evaluate offspring fitness and add to population
      
end while
Compute output from fittest chromosome
ALGORITHM 1 Genetic Algorithm.

5.2 Initial Population

There are various methods for initialization of genetic algorithms, Paul et al. [PRB13] discusses and compares different initialization methods of genetic algorithms for the Traveling Salesman Problem, where the chromosome definition is similar to ours. Based on their comparison and the properties of our problem, we chose to use a gene bank [WHG07], i.e. for each source landmark we compute a subset of target landmarks that are a potential match.

To compare between landmarks on the two shapes we use a descriptor based distance, where we chose the Wave Kernel Signature (WKS) [ASC11]. While this choice can induce some isometric bias, as we generate multiple matches for each source landmark, this bias does not affect our results. Let denote the normalized WKS distance between two landmarks , such that the distance range is , and is normalized separately for each landmark.

Gene bank.

The gene bank of a landmark , denoted by is the set of genes that match to a landmark which is close to it in WKS distance, and is of the same origin. Specifically:

(11)

where we take .

Prominent landmark.

A landmark is denoted as a prominent landmark if its gene bank is not empty, and has at most genes. This indicates that our confidence in this landmark is relatively high, and we will prefer to start the chromosome building process from such landmarks.

Finally, to add more genes to an existing chromosome, we will need the following definition.

Closest matched/unmatched pair.

Given two disjoint subsets , we define the closest matched/unmatched pair as the closest adjacent pair, where each landmark belongs to a different subset. Explicitly:

(12)

5.2.1 Chromosome construction

Using these definitions, we can now address the construction of a new chromosome (see Algorithm 2).

First, we randomly select a prominent landmark , and add a random gene from its gene bank to . We maintain two subsets , that denote the landmarks that have non-empty genes in , and the unprocessed landmarks, respectively. Hence, initially, , and .

Then, we repeatedly find a closest matching pair , and try to add an adjacency preserving gene for which keeps the chromosome valid. First, we look for an AP gene in the gene bank . If none is found, we try to construct an AP gene , where , and is of the same type (i.e., maxima, minima or center) as .

If no AP gene can be constructed which maintains the validity of , an empty gene for is added to , and we look for the next closest matching pair. If is empty, namely, no more adjacent landmarks remain unmatched, empty genes are added to for all the remaining landmarks in .

5.2.2 Match size

The number of non-empty genes in a chromosome, i.e., , implies how many landmarks are used for computing the dense map. The maximal number of non-empty genes is .

On the one hand, if is too small, it is unlikely that the computed map would be useful. Thus, we restrict the minimal match size to , and discard the constructed chromosome if it does not have enough non-empty genes.

On the other hand, we would like to have a variety of possible maps, and thus we force the maximal number of matches to vary in size. Specifically, before a chromosome is constructed, we randomly select a match size . If the chromosome assigns more than landmarks, we remove landmarks from until we reach the required size. The removed landmarks are chosen randomly from the centers class, since we do not want to lose the salient extremities landmarks.

5.2.3 Population construction

We construct chromosomes as described previously until the initial population contains chromosomes [RG02], or a maximal iteration is reached. During construction, repeated chromosomes are discarded, as are chromosomes whose fitness energy . We take the maximal energy threshold to be , which is large enough to allow diverse chromosomes of different quality to be part of the population, yet low enough to discard chromosomes that are very far from a meaningful correspondence.

input : Landmark adjacencies , Landmark origins
output : A chromosome
Pick a random prominent landmark ;
  // seed landmark
Add a random gene from to ;
  // first gene from gene bank
;
  // initialize sets
while  do // Adjacent unmatched landmarks exist
       Find closest matched/unmatched pair
       = ;
        // adjacent gene
       = pickGene() ;
        // gene to add
       add to
       remove from ;
        // landmark was processed
       if  is not empty then
             add to
       end if
      
end while
add empty genes for all ;
  // remaining unmatched landmarks
Function pickGene() is
       foreach  do
             =
             if  is AP to and is valid then
                   return
                  
             end if
            
       end foreach
      return empty gene
      
end
ALGORITHM 2 Create a random chromosome.

5.3 Fitness

Functional map optimization.

The fitness of a chromosome is computed by first using its match to extract a functional map . In fact, since the non-empty genes define a permutation mapping the subset to and vice versa, we use to compute functional maps in both directions. Specifically, we define:

(13)

where are defined in equations (3) and (4), respectively, and we take , .

Given a chromosome , we first compute

(14)

where , and . Note that this computation is very efficient, as these are two unconstrained linear least squares problems.

Functional map refinement.

The optimized functional maps, , can be efficiently refined by converting them to pointwise maps and back, such that they better represent valid pointwise maps. Specifically, we solve:

(15)

where

is a binary row-stochastic matrix, and

is given in Equation (2). This problem can be solved efficiently using a kd-tree, by finding nearest neighbors in . Finally, the optimal functional maps in both directions are given by:

(16)

where is given in Equation (1).

Functional map fitness.

Finally, we evaluate the fitness of the maps with the reversible elastic energy:

(17)

where are defined in section 2.4. Throughout the paper we use a constant . While this objective is highly non-linear and non-convex, the fitness is never optimized directly, but only evaluated during the genetic algorithm, which is well suited for such complex objective functions.

To be precise, the term fitness commonly describes a value that is higher for better chromosomes, while in our case the energy is lower for better chromosomes. Thus, when we write fittest we refer to the chromosome with the lowest energy .

5.4 Selection

In this process, individuals from the population are selected in order to pass their genes to the next generation. At each stage half of the population is selected to mate and create offspring. In order to select the individuals for mating we use a fitness proportionate selection (further discussed in [Bac96]). In our case the probability to select an individual for mating is proportional to 1 over its fitness so that the fitter individuals have a better chance of being selected.

5.5 Crossover

The chromosomes selected for the next generation have a probability to undergo a crossover operation. The crossover operator merges two different input chromosomes, into two new chromosomes, . To combine the input chromosomes in a geometrically consistent way, we again use adjacency preserving genes, as defined in section 5.1. The crossover algorithm (Algorithm 3), is very similar to the initial chromosome creation. Here, however, the source of AP genes comes from the parent chromosomes if possible, and only if these cannot be added while keeping the chromosomes valid, do we consider the gene banks. If no more adjacent unassigned landmarks are available, we pick a random chromosome from the parent that does not invalidate the child chromosome.

input : Landmark adjacencies , chromosomes
output : Chromosomes
;
  // parents
Pick a random landmark ;
  // seed landmark
foreach  do
       Copy the gene from to ;
        // first gene
       ;
        // initialize sets
       while  do // unmatched landmarks exist
             if  then // Adjacent landmarks exist
                   Find closest matched/unmatched pair
                   = ;
                    // adjacent gene
                   = ;
                    // parent genes
                   = pickGene() ;
                    // gene to add
                  
            else
                   = random gene from such that is valid
                   if  then // no more parent genes
                         break
                        
                   end if
                   = source landmark of
                  
             end if
            add to
             remove from ;
              // landmark was processed
             if  is not empty then
                   add to
             end if
            
       end while
      
end foreach
add empty genes for all ;
  // remaining unmatched landmarks
ALGORITHM 3 Crossover.
Figure 3: Convergence of the genetic algorithm. (left) The energy of the fittest offspring and the average energy of the population decrease until convergence indicating the improvement throughout the generation. (right) The fittest offspring at different generations, improving until reaching the right match between the shapes.

5.6 Mutation

After descendants are computed using the crossover operator, they can undergo different mutations, with varying probabilities . Let be the chromosome that undergoes mutation. We define three different types of mutation operators (inspired by [BM05]):

5.6.1 Growth.

With probability . Go over all the empty genes in random order and try to replace one, , by assigning it a corresponding landmark using the following two options.

FMap.

Compute , using equations (14) and (15) and set to the closet landmark on to . Add to if is valid.

Gene Bank.

If the previous attempt failed, set to a random gene from the gene bank , and add it to if is valid.

5.6.2 Shrinkage.

With probability . Randomly select centers landmarks from . Replace each one of the corresponding genes with an empty gene, such that new chromosomes are obtained. Each of these has a single new empty gene. The result is the fittest chromosome among them (including the original before shrinking). We take .

5.6.3 Functional map guidance.

With probability . Compute , using equations (14) and (15). For each landmark set to the closet landmark on to . If injectivity is violated, i.e. , if is a center and is in another category, set and as computed by the functional map guidance. Otherwise, randomly keep one of them and set the other one to 0. We prioritize maxima and minima as they are often more salient.

5.7 Convergence

We stop the iterations when the fittest chromosome remains unchanged for 70 iterations, and its fitness is below a constant value we define as a good energy for a smooth functional map. Alternatively, we stop when a maximal iteration number is reached.

6 Results

6.1 Datasets and Evaluation

Our method computes a sparse correspondence and a functional map that can be used as input to existing semi-automatic methods such as [ESBC19], that we use in this paper. We demonstrate the results of our method on two datasets with different properties.

The SHREC’07 dataset [GBP07] contains a variety of non-isometric shapes as well as ground truth sparse correspondence between manually selected landmarks. This dataset is suitable to demonstrate the advantages of our method, since we address the highly non-isometric case. The recent dataset SHREC’19 [DSLR19] contains shapes of the same semantic class but different topologies, that we use to demonstrate our results in such challenging cases.

Quantitatively evaluating sparse correspondence on this dataset is challenging, since the given sparse ground truth does not necessarily coincide with the computed landmarks. We therefore quantitatively evaluate the results after post processing, where we use the same post processing for all methods (if a method produces sparse correspondence we compute functional maps using these landmarks and run RHM [ESBC19] to extract a pointwise map, and if a method produces a pointwise map we apply RHM directly). Since we initialize RHM with a functional map or a dense map, we use the Euclidean rather than the geodesic embedding that was used in their paper, that is only needed when the initialization is very coarse. We use the evaluation protocol suggested by Kim et al. [KLF11], where the x axis is a geodesic distance between a ground truth correspondence and a computed correspondence, and the y axis is the percentage of correspondences with less than x error. We also allow symmetries, as suggested by Kim et al. [KLF11], by computing the error w.r.t. both the ground truth correspondence and the symmetric map (that is given), and using the map with the lower error as ground truth for comparison.

We qualitatively evaluate our results using color and texture transfer. We visualize sparse correspondence by showing corresponding landmarks in the same color. To visualize functional maps we show a smooth function, visualized by color coding on the target mesh, and transfer it to the source using the functional map. We visualize pointwise maps by texture transfer.

6.2 Population Evolution and Convergence

Figure 4: Visualization of the population evolution throughout the genetic algorithm, for the pig-cow example. The chromosomes are embedded in 2D using Multi Dimensional Scaling [Mea92] using a chromosome distance function (see text for details). The first generation is scattered, and the middle and last generations are more clustered. Additionally, the last generation is concentrated further from the first generation, which means the algorithm evolves towards a good solution even if the initial population does not contain many similar chromosomes.

Figure 3 shows the evolution of the population in the pig-cow example. We plot the mean energy of a chromosome as a function of the generation number, as well as the minimal energy (of the fittest chromosome). We also visualize the functional map and the landmarks of the fittest chromosome throughout the algorithm, which is very inaccurate at the first iteration (the pig’s leg is mapped to the cow’s head), and gets more and more accurate as the population evolves, until finally the map is semantic.

Figure 4 visualizes all the chromosomes of the first, middle, and last generation by embedding the chromosomes in 2D. Each point in the image represents a single chromosome, and the color indicates the generation it belongs to. To embed the chromosomes in 2D in a meaningful manner, we define pairwise distance between chromosomes based on geodesic distances between landmarks (normalized by the shape diameter):

The figure shows that the chromosomes are more scattered in earlier generations, and evolve to chromosomes that did not exist in the first generation.

6.3 Quantitative and Qualitative Comparisons

Figure 5: Quantitative comparison between our final results and fully automatic state of the art methods [KKBL15, DML17, Sah18, RPWO18] (after applying the same post processing [ESBC19] on all the methods to extract a dense map), using the evaluation protocol suggested by Kim et al. [KLF11]. The graph shows that our results are closer to the ground truth than other methods.
Figure 6: Qualitative comparison between the results of our genetic algorithm, i.e. a sparse correspondence and a functional map, and the results of automatic state of the art methods [KKBL15, DML17, Sah18] for sparse correspondence. The functional map is visualized by color transfer (see text for details), and the sparse correspondence is visualized by landmarks with corresponding colors.
Figure 7: Qualitative comparison between our final results and fully automatic state of the art methods [KKBL15, DML17, Sah18, RPWO18] (after applying the same post processing [ESBC19] on all the methods to extract a dense map). The pointwise maps are visualized using texture that is computed on the target mesh (left) and transferred to the source using each method.

We compare our method with a few fully automatic methods for sparse correspondence: "Tight Relaxation" by Kezurer et al. [KKBL15], DS++ by Dym et al. [DML17], and the recent method by Sahillioglu et al. [Sah18] that also used a genetic algorithm (GA+AS, AS stands for Adaptive Sampling which they use for improved results). We apply the same post processing we used for our method on the output sparse correspondence of the other methods, i.e. we compute a functional map as described in section 5.3 and use it as input to the sparse-to-dense post processing method [ESBC19].

Figure 6 visualizes the output sparse correspondence of each of these methods on shapes from SHREC’07 [GBP07], as well as the functional maps that are visualized by transferring smooth functions and color coding. Our method consistently generates semantic results on shapes from various classes.

After the post processing that extracts a dense pointwise map from the functional maps, we can quantitatively compare the results using the protocol suggested by Kim et al. [KLF11] as described in section 6.1. We compare the dense pointwise maps to a recent method by Ren et al. [RPWO18] (BCICP), whose output is a dense map. Since Ren et al. [RPWO18] computes vertex-to-vertex maps, we apply the post processing [ESBC19] on their results as well (BCICP+RHM).

The results are shown in Figure 5, where we used about 100 pairs of shapes from SHREC’07 [GBP07] from 10 different classes. Our method outperforms all previous results. In addition we visualize the final dense maps using texture transfer in Figure 7.

6.4 Correspondence between Shapes of Different Genus

Figure 8: Our results on shapes of genus 1 (cups) and two shapes with different genus.

Our method is not restricted to genus zero, and works for shapes of different topology as well. Figure [ shows our results for two shapes of hands from SHREC’19 [DSLR19]. The left hand shape is of genus 2 (the tips of the index finger and the thumb are connected, as well as the middle of the index and the middle finger), and the right hand shape is of genus 1 (the upper part of the middle and ring finger are connected). Figure 8 shows our sparse correspondence and functional map for two shapes of cups from SHREC’07 [GBP07], both genus 1, and two human shapes from SHREC’19 [DSLR19] with genus 0 and genus 1. Note that in all these cases, despite the difficult topological issues, our approach found a meaningful map.

7 Conclusion

We present an approach for computing an automatic correspondence between shapes of the same semantic class which are non-isometric. We leverage a genetic algorithm for the combinatorial optimization of a small set of automatically computed landmarks, and use a fitness energy that is based on extending the sparse landmarks to a functional map. As a result, we can achieve meaningful maps, that outperform existing state-of-the-art automatic methods, and can additionally handle difficult cases of different topology of the source and target.

We believe that our approach can be generalized in a few ways. First, the decomposition of the automatic mapping computation problem into combinatorial and continuous problems mirrors other tasks in geometry processing which were handled in a similar manner, such as quadrangular remeshing. It is interesting to investigate whether additional analogs exist between these seemingly unrelated problems. Furthermore, it is intriguing to consider what other problems in shape analysis can benefit from genetic algorithms. One potential example is map synchronization for map collections, where the choice of cycles to synchronize is also combinatorial.

References

  • [AL15] Aigerman N., Lipman Y.: Orbifold Tutte embeddings. ACM Transactions on Graphics (TOG) 34 (2015).
  • [AL16] Aigerman N., Lipman Y.: Hyperbolic orbifold Tutte embeddings. ACM Transactions on Graphics (TOG) 35 (2016).
  • [APL15] Aigerman N., Poranne R., Lipman Y.: Seamless surface mappings. ACM Transactions on Graphics (TOG) 34, 4 (2015), 72.
  • [ASC11] Aubry M., Schlickewei U., Cremers D.: The wave kernel signature: A quantum mechanical approach to shape analysis. In

    International Conference on Computer Vision Workshops (ICCV Workshops)

    (2011), IEEE.
  • [Auw07] Auwatanamongkol S.: Inexact graph matching using a genetic algorithm for image recognition. Pattern Recognition Letters 28, 12 (2007), 1428–1437.
  • [Bac96] Back T.: Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press, 1996.
  • [BLM95] Bhanu B., Lee S., Ming J.: Adaptive image segmentation using a genetic algorithm. IEEE Transactions on systems, man, and cybernetics 25, 12 (1995), 1543–1567.
  • [BM05] Brie A. H., Morignot P.: Genetic planning using variable length chromosomes. In ICAPS (2005), pp. 320–329.
  • [BPGK06] Botsch M., Pauly M., Gross M. H., Kobbelt L.: PriMo: coupled prisms for intuitive surface modeling. In Proc. Eurographics Symposium on Geometry Processing (2006), pp. 11–20.
  • [CMS18] Cheng X., Mishne G., Steinerberger S.:

    The geometry of nodal sets and outlier detection.

    Journal of Number Theory 185 (2018), 48–64.
  • [CTL04] Chow C. K., Tsui H. T., Lee T.: Surface registration using a dynamic genetic algorithm. Pattern recognition 37, 1 (2004), 105–117.
  • [CWH97] Cross A. D., Wilson R. C., Hancock E. R.: Inexact graph matching using genetic search. Pattern Recognition 30, 6 (1997), 953–970.
  • [dBDFN16] de Buhan M., Dapogny C., Frey P., Nardoni C.: An optimization method for elastic shape matching. C. R. Math. Acad. Sci. Paris 354, 8 (2016), 783–787.
  • [DML17] Dym N., Maron H., Lipman Y.: Ds++: A flexible, scalable and provably tight relaxation for matching problems. ACM Transactions on Graphics (TOG) 36, 6 (2017).
  • [DSLR19]

    Dyke R., Stride C., Lai Y.-K., Rosin P. L.

    :
    Shrec 2019. https://shrec19.cs.cf.ac.uk/, 2019.
  • [EBC17] Ezuz D., Ben-Chen M.: Deblurring and denoising of maps between shapes. In Computer Graphics Forum (2017), vol. 36, Wiley Online Library, pp. 165–174.
  • [EHA19] Ezuz D., Heeren B., Azencot O., Rumpf M., Ben-Chen M.: Elastic correspondence between triangle meshes. In Computer Graphics Forum (2019), vol. 38.
  • [ERGB16] Eynard D., Rodola E., Glashoff K., Bronstein M. M.: Coupled functional maps. In 2016 Fourth International Conference on 3D Vision (3DV) (2016), IEEE, pp. 399–407.
  • [ESBC19] Ezuz D., Solomon J., Ben-Chen M.: Reversible harmonic maps between discrete surfaces. ACM Transactions on Graphics (2019).
  • [ESKBC17] Ezuz D., Solomon J., Kim V. G., Ben-Chen M.: Gwcnn: A metric alignment layer for deep shape analysis. In Computer Graphics Forum (2017), vol. 36, pp. 49–57.
  • [GBKS18] Gehre A., Bronstein M., Kobbelt L., Solomon J.: Interactive curve constrained functional maps. In Computer Graphics Forum (2018), vol. 37, Wiley Online Library, pp. 1–12.
  • [GBP07] Giorgi D., Biasotti S., Paraboschi L.: SHREC: Shape retrieval contest: Watertight models track, 2007.
  • [HO17] Huang R., Ovsjanikov M.: Adjoint map representation for shape analysis and matching. In Computer Graphics Forum (2017), vol. 36, pp. 151–163.
  • [Hol92] Holland J. H.: Genetic algorithms. Scientific american 267, 1 (1992), 66–73.
  • [HRS14] Heeren B., Rumpf M., Schröder P., Wardetzky M., Wirth B.: Exploring the geometry of the space of shells. Comput. Graph. Forum 33, 5 (2014), 247–256.
  • [HRWW12] Heeren B., Rumpf M., Wardetzky M., Wirth B.: Time-discrete geodesics in the space of shells. Comput. Graph. Forum 31, 5 (2012), 1755–1764.
  • [IRS18] Iglesias J. A., Rumpf M., Scherzer O.: Shape-aware matching of implicit surfaces based on thin shell energies. Found. Comput. Math. 18, 4 (2018), 891–927.
  • [KKBL15] Kezurer I., Kovalsky S. Z., Basri R., Lipman Y.: Tight relaxation of quadratic matching. In Computer Graphics Forum (2015), vol. 34, pp. 115–128.
  • [KLF11] Kim V. G., Lipman Y., Funkhouser T.: Blended Intrinsic Maps. ACM Transactions on Graphics (TOG) 30 (2011).
  • [LDRS05] Litke N., Droske M., Rumpf M., Schröder P.: An image processing approach to surface matching. In Proc. Eurographics Symposium on Geometry Processing (2005), pp. 207–216.
  • [MB00] Maulik U., Bandyopadhyay S.: Genetic algorithm-based clustering technique. Pattern recognition 33, 9 (2000), 1455–1465.
  • [MCSK17] Mandad M., Cohen-Steiner D., Kobbelt L., Alliez P., Desbrun M.: Variance-minimizing transport plans for inter-surface mapping. ACM Transactions on Graphics 36 (2017), 14.
  • [MDK16] Maron H., Dym N., Kezurer I., Kovalsky S., Lipman Y.: Point registration via efficient convex relaxation. ACM Transactions on Graphics (TOG) 35, 4 (2016).
  • [MDW08] Munsell B. C., Dalal P., Wang S.: Evaluating shape correspondence for statistical shape analysis: A benchmark study. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (2008), 2023–2039.
  • [Mea92] Mead A.: Review of the development of multidimensional scaling methods. Journal of the Royal Statistical Society: Series D (The Statistician) 41, 1 (1992), 27–39.
  • [NO17] Nogneng D., Ovsjanikov M.: Informative descriptor preservation via commutativity for shape matching. In Computer Graphics Forum (2017), vol. 36, pp. 259–267.
  • [OBCS12] Ovsjanikov M., Ben-Chen M., Solomon J., Butscher A., Guibas L.: Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (TOG) 31, 4 (2012).
  • [OCB16] Ovsjanikov M., Corman E., Bronstein M., Rodolà E., Ben-Chen M., Guibas L., Chazal F., Bronstein A.: Computing and processing correspondences with functional maps. In SIGGRAPH ASIA 2016 Courses (2016), ACM, p. 9.
  • [OM97] Ozcan E., Mohan C. K.: Partial shape matching using genetic algorithms. Pattern Recognition Letters 18, 10 (1997), 987–992.
  • [PBDSH13] Panozzo D., Baran I., Diamanti O., Sorkine-Hornung O.: Weighted averages on surfaces. ACM Transactions on Graphics (TOG) 32, 4 (2013), 60.
  • [PRB13] Paul P. V., Ramalingam A., Baskaran R., Dhavachelvan P., Vivekanandan K., Subramanian R., Venkatachalapathy V.: Performance analyses on population seeding techniques for genetic algorithms. International Journal of Engineering and Technology (IJET) 5, 3 (2013), 2993–3000.
  • [RG02] Rylander S. G. B., Gotshall S.: Optimal population size and the genetic algorithm. Population 100, 400 (2002), 900.
  • [RPWO18] Ren J., Poulenard A., Wonka P., Ovsjanikov M.: Continuous and orientation-preserving correspondences via functional maps. ACM Transactions on Graphics (TOG) 37, 6 (2018).
  • [SA07] Sorkine O., Alexa M.: As-rigid-as-possible surface modeling. In Proc. Eurographics Symposium on Geometry Processing (2007), pp. 109–116.
  • [Sah18] Sahillioğlu Y.: A genetic isometric shape correspondence algorithm with adaptive sampling. ACM Trans. Graph. 37, 5 (Oct. 2018).
  • [SBB05] Silva L., Bellon O. R. P., Boyer K. L.: Precision range image registration using a robust surface interpenetration measure and enhanced genetic algorithms. IEEE transactions on pattern analysis and machine intelligence 27, 5 (2005), 762–776.
  • [SP04] Sumner R. W., Popović J.: Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23 (2004).
  • [SPKS16] Solomon J., Peyré G., Kim V. G., Sra S.: Entropic metric alignment for correspondence problems. ACM Transactions on Graphics (TOG) 35, 4 (2016), 72.
  • [TCL13] Tam G. K., Cheng Z.-Q., Lai Y.-K., Langbein F. C., Liu Y., Marshall D., Martin R. R., Sun X.-F., Rosin P. L.: Registration of 3d point clouds and meshes: a survey from rigid to nonrigid. IEEE Transactions on Visualization and Computer Graphics 19, 7 (2013), 1199–1217.
  • [UM93] Unger R., Moult J.: Genetic algorithms for protein folding simulations. Journal of molecular biology 231, 1 (1993), 75–81.
  • [VKZHCO11] Van Kaick O., Zhang H., Hamarneh G., Cohen-Or D.: A survey on shape correspondence. In Computer Graphics Forum (2011), vol. 30, pp. 1681–1707.
  • [VLR17] Vestner M., Litman R., Rodola E., Bronstein A., Cremers D.:

    Product manifold filter: Non-rigid shape correspondence via kernel density estimation in the product space.

    In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), IEEE, pp. 6681–6690.
  • [WHG07] Wei Y., Hu Y., Gu K.: Parallel search strategies for tsps using a greedy genetic algorithm. In Third International Conference on Natural Computation (ICNC 2007) (2007), vol. 3, IEEE, pp. 786–790.
  • [WSSC11] Windheuser T., Schlickewei U., Schmidt F. R., Cremers D.:

    Geometrically consistent elastic matching of 3d shapes: A linear programming solution.

    In Proc. IEEE International Conference on Computer Vision (2011), pp. 2134–2141.
  • [YAF99] Yamany S. M., Ahmed M. N., Farag A. A.: A new genetic-based technique for matching 3-d curves and surfaces. Pattern Recognition 32, 10 (1999), 1817–1820.