1 Introduction
Shape correspondence between a pair of surfaces is the task of estimating a semantically meaningful map from one surface onto the other. In geometry processing, surfaces are typically represented as compact twodimensional Riemannian manifolds. Given two such surfaces
and , the correspondence map is typically obtained as the solution to an optimization problem where the notion of a good or bad correspondence is explicitly modeled by the objective function.Typically, such cost functions comprise of a combination of two components: pointwise descriptor preservation or function preservation constraints, and pairwise descriptor preservation or operator commutativity constraints. A pointwise descriptor associates a scalar value to every point on the surface. Therefore, any candidate for its corresponding point on the second shape must have an equivalent descriptor value. For a surface discretized by
points, a single pointwise descriptor is a vector of size
. Notable examples are the heat kernel signature (HKS) [21], the wave kernel (WKS) [4], shape context signatures [12] etc.In contrast, a pairwise descriptor associates a scalar value for every pair of points on each shape. Therefore, given a pair of points on the first shape, a correspondence map must result in choosing corresponding pairs of points on the second shape yielding equivalent pairwise descriptor values. For an point surface, a single pairwise descriptor is a matrix of size . Typical examples include, intergeodesic distance matrices [7] and heat kernels [18, 24] of the surfaces. Pairwise descriptors can also be interpreted as operators acting on functions defined over the surface. Therefore the constraint of preserving a pairwise descriptor is equivalent to the argument that applying the operator to any function and transferring it by correspondence must be equivalent to transferring the function first and then operating on it, thus commuting with the correspondence map.
The functional maps paradigm [17], provides for an elegant linear algebraic method of estimating correspondence maps between surfaces. Under the assumption of approximate isometry, the main idea is to formulate the pairwise and pointwise constraints in the spectral domain of the shapes and represent the map using the eigenfunctions of the laplace beltrami operators of the surfaces. This formulation enables replacing a combinatorial nonconvex optimization problem of finding permutations with a simple linear least squares estimation yielding a linear map between the functional spaces of the pair of manifolds.
In most practical applications, it is usually convenient to find reliable pointwise descriptors like point or part correspondences. Eg: shape segments [15], spectral descriptors [21, 4, 14], histograms of local features [23] etc. that are relatively easier to identify and compute. The authors of [16] demonstrate that the information from a pointwise descriptor can be extended to construct a linear operator from it. They propose an operator (or pairwise descriptor) that acts on a function through a pointwise multiplication of the descriptor. Imposing a commutativity constraint with this operator leads to an improved functional map estimation from the same set of known corresponding pointwise descriptors.
Advancing this idea of constructing a pairwise operator from a pointwise descriptor, we explore a novel operator which we call the bilateral operator. The bilateral operator is a smoothing operator, where the weights of averaging are obtained by a combination of intrinsic and descriptor similarities. Each pointwise descriptor can be used to define a pairwise bilateral operator whose action indicates descriptor dependent smoothing of a function over the manifold. By requiring that the correspondence map commute with such an operator, we show a considerable increase in correspondence quality in comparison to previous functional map approaches. The intuition behind our method is that combination of geodesic and descriptor closeness yields high quality correspondence. This means that pairs of points on one shape that are close geodesically or have similar descriptor values correspond very distinctly to pairs of such similar points on the other shape, given the availability of good quality corresponding pointwise descriptors.
2 Background
Without loss of generality, let us consider a pair of shapes and that are sampled consistently by points each. We consider the discrete counterpart of the correspondence map and observe that it can be represented as a permutation matrix satisfying . where is a column vector of ones. A majority of the shape correspondence algorithms can be expressed as an energy minimization problem having the following form [24]:
(1) 
where denotes the space of permutation matrices.
Given a set of corresponding pointwise descriptors: , the pointwise descriptor preservation term can be written as:
(2) 
and similarly, for a set of corresponding pairwise descriptors , the pairwise descriptor preservation (or operator commutativity constraint) is given by:
(3) 
The functional maps paradigm [17] allows for solving this problem efficiently by leveraging the knowledge of a compact basis for representing functions on each surface. Typically, the eigenfunctions of the Laplace Beltrami operator of each surface are used, since they form a convenient basis to represent functions and more importantly, they are proven to be optimal for representing smooth functions over manifolds [1].
To fix notations, let be a twodimensional Riemannian manifold. Consider the space of functions defined on the manifold: . The positive semidefinite LaplaceBeltrami operator (LBO) generalizes the notion of the Laplacian operator from Euclidean spaces to surfaces. admits an orthonormal eigendecomposition
, where the eigenvalues form a discrete spectrum
and the eigenfunctions } form an orthonormal basis for , thereby allowing us to expand any smooth function as a Fourier series(4) 
Let and be the corresponding eigenvalues and eigenfunctions of the LBO on . Then the functional representation of the correspondence map in the spectral basis is given by
(5) 
Where the matrix C is a functional maps matrix, whose elements encode the coefficients of the correspondence map represented in the spectral domain. The main utility of the functional maps framework is that, a much larger permutation can be approximated by a smaller matrix C by using a few low frequency eigenfunctions of the laplace beltrami operators and respectively.
By rewriting equation 1 using the spectral approximation of 5, the functional maps matrix can be obtained as the solution to the problem:
(6)  
where
(7)  
(8)  
(9) 
Where and
are the area matrices of the respective shapes. Different shape correspondence algorithms in literature can be viewed as particular cases of the loss functions
1 or 6, depending on whether the correspondence was solved in the spatial domain as a permutation or in the spectral domain as a functional map. Notable examples include [16], which proposes a diagonal operator whos action is equivalent to a pointwise multiplication of the corresponding descriptors and . Similarly [9, 8, 2] show the application of geodesic distances and [24] proposed the use of heatkernels of the surfaces as pairwise descriptors.(a) A gaussian kernel centered around a point on the belly. (b)(e) second  fifth eigenfunctions of the kernel operator. The colormap has been deliberately tresholded to convey the harmonic progression of the eigenvectors in the descriptor space. This shows a sequential clustering of regions with equal descriptor values. Enforcing the correspondence to commute with this kernel is equivalent to the additional constraint that these regions must also be preserved by the correspondence map, in addition to a straightforward pointwise preservation of the pointwise descriptor.
3 Bilateral Operators
Our main observation is that we can formulate a more informative pairwise constraint on the correspondence map from an input of pointwise descriptors. This means that in addition to requiring that the correspondence map preserve the descriptor values pointwise as in equation 2, the correspondence map must also preserve the similarities between every pair of points as measured by the descriptor itself. For a given descriptor and a pair of points , we can associate a notion of similarity between them given by:
(10) 
Equation 10 is an operator constructed from a pointwise descriptor . One can also interpret it as a smoothing operator over the surface where the weights of averaging are determined by the kernel in equation 10.
A more intrinsic notion of similarity over the surface is given by the heat kernel. Heat kernels are fundamental solutions to the heat diffusion equation over the manifold.
(11) 
with the initial conditions . Here represents the amount of heat at point at time . The solution to is given by:
(12) 
The heat kernel can be represented in terms of the spectral decomposition of the LBO and is given by:
(13) 
For every pair of pointwise descriptors , we can compute the corresponding bilateral operators that combine the descriptor similarity measure of 10 with an intrinsic notion of similarity encoded by the heat kernel:
(14) 
The bilateral operators of equation 14 are conceptually related to a promonient filtering technique used for the problem of image denoising. Image denoising is the problem of recovering an image signal from its noisy estimate. As in the case of any noise removal problem, the common intuition is to employ some form of smoothing to the image in which the value of the image function at each pixel is a weighted average of its neighbors. The classical signal processing view of smoothing employs a gaussian weighted averaging of the the image as in equation 15.
(15) 
Here encodes the 2D spatial location of pixel , and refers to the pixel indices in the spatial neighborhood of the pixel. is the estimate of the denoised image. The operation of Equation 15 is equivalent to a heat diffusion of on the standard 2D euclidean plane with the noisy image as the initial condition. This is validated by the fact that the heat kernel in planar domain is the standard gaussian kernel:
(16) 
However, an important property desirable from such smoothing operators is edge preservation. See figure 1. A gaussian smoothing of equation 15 will smooth away essential details of the image like dominant edges and textures, since the weights are independent of image intensity. This motivated certain edge preserving filters that allowed for a imagedependent heat diffusion. The beltrami flow framework [19, 20] and the the bilateral filter [22] introduced an additional component that weights a pair of pixels by the difference in their image values in addition to geometric closeness ensured by the heat kernel. Therefore the action of the bilateral filter was to average pixels that were close in the geometric and functional domain, leading to an improved image denoising filter.
(17) 
In this paper, we assert that a similar argument can be extended to the recovery of correspondence maps. Given the knowledge of good quality pointwise descriptors, we assert that a desirable correspondence map must preserve a combination of both geodesic and descriptor closeness. This can be achieved by commuting with the bilateral operators of equation 14.
There is however a minor conceptual difference between the bilateral operators used in image smoothing and the operators of equation 14. An edge preserving kernel for image smoothing demands that the points be spatially close and also have similar image intensities. Hence the resulting bilateral operator is a pointwise multiplication of the kernels (or a Hadamard multiplication of two positive definite matrices). However, this property is appropriate for a denoising scenario where both similarities need to be strong for the final weight to be significant.
In contrast, the bilateral operator proposed in this paper suggests an additive combination of geodesic and descriptor similarity. Hence either of the similarities need to be strong for the final kernel value to be significant. The constraint of commuting with such an operator leads to the following observation. Pairs of points on the source shape that are similar in a geodesic sense (hence a higher value for the heat kernel) or by way of having identical descriptor values must correspond distinctly to similar pairs on the target. The optimal correspondence ignores only those pairs that are neither geodesically close nor have similar descriptor values. Intuitively, this policy works favourably when the shapes are not strictly related by an isometry, since there exist some pairs of points that are relatively far apart on the surface but worthy of good matching. The heat kernel may not weight such pairs highly. However, such pairs can be rightly deemed close by an informative pointwise descriptor and this relationship is leveraged by the pairwise bilateral operator. Hence commuting with the bilateral operator enables a more informative utilization of each descriptor leading to a better correspondence quality for a smaller budget of pointwise descriptors.
Figure 2 provides a visualization on the action of the kernel operator of equation 10. The eigenfunctions show a harmonic progression in the descriptor space, that is, the shape is harmonically clustered into regions of similar descriptor values. This is in contrast to the global manifold harmonics that are the eigenfunctions of the heat kernel. Therefore, given that we have knowledge of a pair of corresponding descriptors, we require that, not only must the correspondence map preserve the value of the descriptor pointwise, but also the layered pairwise relationship between every pair of points as encoded by the descriptor centric kernel of Equation 10. This philosophy works best when we combine it with a geometric notion of similarity encoded in the heat kernel.
3.1 Numerical Computation
For each descriptor, an explicit construction of the bilateral operators demands the population of a matrix using equation 10. However, for a functional map estimation, we only need the spectral equivalents of the operators for its use in the optimization 6. These can be computed as follows:
(18) 
For spectral projection of the descriptor kernels and , we use the nystrom extension for approximating large positive definite matrices [26]. In particular, the Nystrom approximation requires only a sparse set of columns of any kernel matrix. Let be such a column stacked matrix of the columns of . Let be the smaller intersecting submatrix of . Then we have the following expressions:
(19)  
(20) 
and a similar expression may be obtained for the kernel matrices for shape . We use the farthest point sampling strategy [11] for constructing the column matrix . By evaluating the kernel functions 10 for only a sparse subsampled set of points and use the approximations of equation 20 we arrive at the spectral coefficients for the pairwise bilateral operators to be used in a functional map pipeline.
4 Correspondence Algorithm
Input:  Source and Target shapes , 

spectral geometries of shapes :  
Output:  Functional Map matrix C, 
correspondence map 

Find pairs of pointwise descriptors for each shape and compute their spectral equivalents

From each pair, construct the corresponding bilateral operators using the computational routine outlined in section 3.1

Incorporate constraints into a linear system and solve for functional map matrix C by minimizing equation 6

Refine the map using an ICP like iteration [17]

Compute a point to point correspondence using the refined map
5 Experiments
The general philosophy of our experiments is to compare between various functional map algorithms by imputing all of them with the
same set of descriptors and using the same linear optimizer with identical coefficients for the pairwise and pointwise constraints. We compare with three algorithms: the original functional maps framework [17] without any descriptor based regularizer, the diagonal operator proposed in [16] and kernel functional maps [25]. All of these algorithms have the same objective of maximizing the information from a fixed budget of pointwise descriptors.First, we conduct a synthetic experiment to assess the variation in correspondence quality as a function of number of perfect input pointwise descriptors over 25 shape pairs of FAUST [5]
. The objective of this experiment is to assess which functional map algorithm (or constraint) maximally exploits the information from a fixed budget of high quality descriptors. Most recent and arguably stateoftheart algorithms on shape correspondence are tailored towards learning optimal descriptors with properties of localization, sensitivity and invariance. Recent results from deeplearning frameworks have demonstrated that such descriptors can be learned from examples
[14, 6, 13, 10]. What we intend to answer with the first experiment is the question, Given that we have highquality pointwise descriptors, which functional map algorithm achieves the best correspondence quality?We compute descriptors by constructing gaussian kernels over groundtruth corresponding points (see Figure 2 (a)), and progressively increase the number of such descriptors as the input to the different algorithms.
(21) 
What we expect to see is a decay in the average correspondence geodesic error as the number of such descriptors increase for each algorithm. We use 60 eigenbasis functions for all shapes algorithms. The parameters for all algorithms we chosen as per the choices made in the respective papers. For the bilateral kernel we chose and and for all experiments. Figure 3 shows the results. We see that for a few descriptors, the bilateral filter shows a much lower average correspondence error for the same number of input descriptors. However, all algorithms eventually converge similarly when the number of descriptors increase. Table 1 visualizes the correspondence for the first few landmarks of the plot.
However, general descriptors are far from perfect. This means that there will be a problem of noise leading to inaccurate correspondence. Therefore, in such scenarios, it is not surprising to expect that increasing the number of descriptors may not necessarily lead to an improved correspondence. Moreover the existence of outliers may even deteriorate correspondence quality.
Figure 6 shows the results of our approach compared to [17],[16],[25] on SCAPE and FAUST datasets. We evaluate each algorithm on randomly 100 chosen pairs, for each shape we used 100 discrete laplacebeltarami eigenfunctions as the functional basis. The results of figures 6 show that the bilateral kernel is very potent at lower number of descriptors, and compares favourably with other algorithms for larger number of descriptors, similar to the results in of figure 3. Figure 5 shows comparison of qualitative results of point to point mapping from the source to a target shape using 10 wave kernel signature/map [4] descriptors. The qualitative results are inline with the quantitative ones, see figure 6.
6 Conclusion
In this paper, we propose a novel operator constructed from a pointwise descriptor. We introduce the bilateral operator that combines a descriptor based similarity with the intrinsic geometry of the surface encoded in the heat kernel. By enforcing commutativity constraints with this operator, we demonstrate that we can achieve a better correspondence quality, given a fixed budget of pointwise descriptors.
References
 [1] Y. Aflalo, H. Brezis, and R. Kimmel. On the optimality of shape and data representation in the spectral domain. SIAM Journal on Imaging Sciences, 8(2):1141–1160, 2015.

[2]
Y. Aflalo, A. Dubrovina, and R. Kimmel.
Spectral generalized multidimensional scaling.
International Journal of Computer Vision
, 118(3):380–392, 2016.  [3] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. Scape: shape completion and animation of people. In ACM transactions on graphics (TOG), volume 24, pages 408–416. ACM, 2005.
 [4] M. Aubry, U. Schlickewei, and D. Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In 2011 IEEE international conference on computer vision workshops (ICCV workshops), pages 1626–1633. IEEE, 2011.

[5]
F. Bogo, J. Romero, M. Loper, and M. J. Black.
Faust: Dataset and evaluation for 3d mesh registration.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pages 3794–3801, 2014. 
[6]
D. Boscaini, J. Masci, E. Rodolà, and M. Bronstein.
Learning shape correspondence with anisotropic convolutional neural networks.
In Advances in Neural Information Processing Systems, pages 3189–3197, 2016.  [7] A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Generalized multidimensional scaling: a framework for isometryinvariant partial surface matching. Proceedings of the National Academy of Sciences, 103(5):1168–1172, 2006.
 [8] A. Dubrovina and R. Kimmel. Matching shapes by eigendecomposition of the laplacebeltrami operator. In Proc. 3DPVT, volume 2, page 2, 2010.
 [9] A. Dubrovina and R. Kimmel. Approximately isometric shape correspondence by matching pointwise spectral features and global geodesic structures. Advances in Adaptive Data Analysis, 3(01n02):203–228, 2011.
 [10] O. Halimi, O. Litany, E. Rodola, A. M. Bronstein, and R. Kimmel. Unsupervised learning of dense shape correspondence. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

[11]
D. S. Hochbaum and D. B. Shmoys.
A best possible heuristic for the kcenter problem.
Mathematics of operations research, 10(2):180–184, 1985.  [12] I. Kokkinos, M. M. Bronstein, R. Litman, and A. M. Bronstein. Intrinsic shape context descriptors for deformable shapes. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 159–166. IEEE, 2012.
 [13] O. Litany, T. Remez, E. Rodolà, A. Bronstein, and M. Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the IEEE International Conference on Computer Vision, pages 5659–5667, 2017.
 [14] R. Litman and A. M. Bronstein. Learning spectral descriptors for deformable shape correspondence. IEEE transactions on pattern analysis and machine intelligence, 36(1):171–180, 2013.
 [15] R. Litman, A. M. Bronstein, and M. M. Bronstein. Stable volumetric features in deformable shapes. Computers & Graphics, 36(5):569–576, 2012.
 [16] D. Nogneng and M. Ovsjanikov. Informative descriptor preservation via commutativity for shape matching. In Computer Graphics Forum, volume 36, pages 259–267. Wiley Online Library, 2017.
 [17] M. Ovsjanikov, M. BenChen, J. Solomon, A. Butscher, and L. Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (TOG), 31(4):30, 2012.
 [18] M. Ovsjanikov, Q. Mérigot, F. Mémoli, and L. Guibas. One point isometric matching with the heat kernel. In Computer Graphics Forum, volume 29, pages 1555–1564. Wiley Online Library, 2010.
 [19] N. Sochen, R. Kimmel, and A. M. Bruckstein. Diffusions and confusions in signal and image processing. Journal of Mathematical Imaging and Vision, 14(3):195–209, 2001.
 [20] A. Spira, R. Kimmel, and N. Sochen. A shorttime beltrami kernel for smoothing images and manifolds. IEEE Transactions on Image Processing, 16(6):1628–1636, 2007.
 [21] J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multiscale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383–1392. Wiley Online Library, 2009.
 [22] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Iccv, volume 98, page 2, 1998.
 [23] F. Tombari, S. Salti, and L. Di Stefano. Unique signatures of histograms for local surface description. In European conference on computer vision, pages 356–369. Springer, 2010.
 [24] M. Vestner, Z. Lähner, A. Boyarski, O. Litany, R. Slossberg, T. Remez, E. Rodola, A. Bronstein, M. Bronstein, R. Kimmel, et al. Efficient deformable shape correspondence via kernel matching. In 2017 International Conference on 3D Vision (3DV), pages 517–526. IEEE, 2017.
 [25] L. Wang, A. Gehre, M. M. Bronstein, and J. Solomon. Kernel functional maps. In Computer Graphics Forum, volume 37, pages 27–36. Wiley Online Library, 2018.
 [26] C. Williams and M. Seeger. Using the nystroem method to speed up kernel machines. Advances in Neural Information Processing Systems 13, 2001.
Comments
There are no comments yet.