 # Bilateral Operators for Functional Maps

A majority of shape correspondence frameworks are based on devising pointwise and pairwise constraints on the correspondence map. The functional maps framework allows for formulating these constraints in the spectral domain. In this paper, we develop a functional map framework for the shape correspondence problem by constructing pairwise constraints using point-wise descriptors. Our core observation is that, every point-wise descriptor allows for the construction a pairwise kernel operator whose low frequency eigenfunctions depict regions of similar descriptor values at various scales of frequency. By aggregating the pairwise information from the descriptor and the intrinsic geometry of the surface encoded in the heat kernel, we construct a hybrid kernel and call it the bilateral operator. Analogous to the edge preserving bilateral filter in image processing, the action of the bilateral operator on a function defined over the manifold yields a descriptor dependent local smoothing of that function. By forcing the correspondence map to commute with the Bilateral operator, we show that we can maximally exploit the information from a given set of pointwise descriptors in a functional map framework.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Shape correspondence between a pair of surfaces is the task of estimating a semantically meaningful map from one surface onto the other. In geometry processing, surfaces are typically represented as compact two-dimensional Riemannian manifolds. Given two such surfaces

and , the correspondence map is typically obtained as the solution to an optimization problem where the notion of a good or bad correspondence is explicitly modeled by the objective function.

Typically, such cost functions comprise of a combination of two components: pointwise descriptor preservation or function preservation constraints, and pairwise descriptor preservation or operator commutativity constraints. A point-wise descriptor associates a scalar value to every point on the surface. Therefore, any candidate for its corresponding point on the second shape must have an equivalent descriptor value. For a surface discretized by

points, a single point-wise descriptor is a vector of size

. Notable examples are the heat kernel signature (HKS) , the wave kernel (WKS) , shape context signatures  etc.

In contrast, a pairwise descriptor associates a scalar value for every pair of points on each shape. Therefore, given a pair of points on the first shape, a correspondence map must result in choosing corresponding pairs of points on the second shape yielding equivalent pairwise descriptor values. For an point surface, a single pairwise descriptor is a matrix of size . Typical examples include, inter-geodesic distance matrices  and heat kernels [18, 24] of the surfaces. Pairwise descriptors can also be interpreted as operators acting on functions defined over the surface. Therefore the constraint of preserving a pairwise descriptor is equivalent to the argument that applying the operator to any function and transferring it by correspondence must be equivalent to transferring the function first and then operating on it, thus commuting with the correspondence map.

The functional maps paradigm , provides for an elegant linear algebraic method of estimating correspondence maps between surfaces. Under the assumption of approximate isometry, the main idea is to formulate the pairwise and pointwise constraints in the spectral domain of the shapes and represent the map using the eigenfunctions of the laplace beltrami operators of the surfaces. This formulation enables replacing a combinatorial non-convex optimization problem of finding permutations with a simple linear least squares estimation yielding a linear map between the functional spaces of the pair of manifolds.

In most practical applications, it is usually convenient to find reliable pointwise descriptors like point or part correspondences. Eg: shape segments , spectral descriptors [21, 4, 14], histograms of local features  etc. that are relatively easier to identify and compute. The authors of  demonstrate that the information from a point-wise descriptor can be extended to construct a linear operator from it. They propose an operator (or pairwise descriptor) that acts on a function through a point-wise multiplication of the descriptor. Imposing a commutativity constraint with this operator leads to an improved functional map estimation from the same set of known corresponding point-wise descriptors.

Advancing this idea of constructing a pairwise operator from a pointwise descriptor, we explore a novel operator which we call the bilateral operator. The bilateral operator is a smoothing operator, where the weights of averaging are obtained by a combination of intrinsic and descriptor similarities. Each pointwise descriptor can be used to define a pairwise bilateral operator whose action indicates descriptor dependent smoothing of a function over the manifold. By requiring that the correspondence map commute with such an operator, we show a considerable increase in correspondence quality in comparison to previous functional map approaches. The intuition behind our method is that combination of geodesic and descriptor closeness yields high quality correspondence. This means that pairs of points on one shape that are close geodesically or have similar descriptor values correspond very distinctly to pairs of such similar points on the other shape, given the availability of good quality corresponding pointwise descriptors.

## 2 Background

Without loss of generality, let us consider a pair of shapes and that are sampled consistently by points each. We consider the discrete counterpart of the correspondence map and observe that it can be represented as a permutation matrix satisfying . where is a column vector of ones. A majority of the shape correspondence algorithms can be expressed as an energy minimization problem having the following form :

 Π∗=argminΠ∈Pnh(Π)+αg(Π) (1)

where denotes the space of permutation matrices.

Given a set of corresponding point-wise descriptors: , the pointwise descriptor preservation term can be written as:

 h(Π)=i=N∑i=1||Πf(i)X−f(i)Y||2 (2)

and similarly, for a set of corresponding pairwise descriptors , the pairwise descriptor preservation (or operator commutativity constraint) is given by:

 g(Π)=i=M∑i=1||ΠO(i)X−O(i)YΠ||2 (3)

The functional maps paradigm  allows for solving this problem efficiently by leveraging the knowledge of a compact basis for representing functions on each surface. Typically, the eigenfunctions of the Laplace Beltrami operator of each surface are used, since they form a convenient basis to represent functions and more importantly, they are proven to be optimal for representing smooth functions over manifolds .

To fix notations, let be a two-dimensional Riemannian manifold. Consider the space of functions defined on the manifold: . The positive semi-definite Laplace-Beltrami operator (LBO) generalizes the notion of the Laplacian operator from Euclidean spaces to surfaces. admits an orthonormal eigendecomposition

, where the eigenvalues form a discrete spectrum

and the eigenfunctions } form an orthonormal basis for , thereby allowing us to expand any smooth function as a Fourier series

 f(x)=∑i≥1⟨ϕi,f⟩Xϕi(x) (4)

Let and be the corresponding eigenvalues and eigenfunctions of the LBO on . Then the functional representation of the correspondence map in the spectral basis is given by

 Π=ΨCΦT (5)

Where the matrix C is a functional maps matrix, whose elements encode the coefficients of the correspondence map represented in the spectral domain. The main utility of the functional maps framework is that, a much larger permutation can be approximated by a smaller matrix C by using a few low frequency eigenfunctions of the laplace beltrami operators and respectively.

By rewriting equation 1 using the spectral approximation of 5, the functional maps matrix can be obtained as the solution to the problem:

 C∗=argminC i=N∑i=1||Cˆf(i)X−ˆf(i)Y||2 (6) +α i=M∑i=1||CˆO(i)X−ˆO(i)YC||2

where

 ˆf(i)X = ΦTATXf(i)X,ˆf(i)Y=ΨTATYf(i)Y (7) ˆO(i)X = ΦTATXO(i)XAXΦ (8) ˆO(i)Y = ΨTATYO(i)YAYΨ (9)

Where and

are the area matrices of the respective shapes. Different shape correspondence algorithms in literature can be viewed as particular cases of the loss functions

1 or 6, depending on whether the correspondence was solved in the spatial domain as a permutation or in the spectral domain as a functional map. Notable examples include , which proposes a diagonal operator whos action is equivalent to a pointwise multiplication of the corresponding descriptors and . Similarly [9, 8, 2] show the application of geodesic distances and  proposed the use of heat-kernels of the surfaces as pairwise descriptors.

## 3 Bilateral Operators

Our main observation is that we can formulate a more informative pairwise constraint on the correspondence map from an input of point-wise descriptors. This means that in addition to requiring that the correspondence map preserve the descriptor values point-wise as in equation 2, the correspondence map must also preserve the similarities between every pair of points as measured by the descriptor itself. For a given descriptor and a pair of points , we can associate a notion of similarity between them given by:

 Kf(i)X(x,x′)=e−(f(i)X(x)−f(i)X(x′))22σ2 (10)

Equation 10 is an operator constructed from a point-wise descriptor . One can also interpret it as a smoothing operator over the surface where the weights of averaging are determined by the kernel in equation 10.

A more intrinsic notion of similarity over the surface is given by the heat kernel. Heat kernels are fundamental solutions to the heat diffusion equation over the manifold.

 ∂u(t,x)∂t=ΔXu(t,x) (11)

with the initial conditions . Here represents the amount of heat at point at time . The solution to is given by:

 u(t,x)=∫XHX(t,x,x′)u0(x′)dx′ (12)

The heat kernel can be represented in terms of the spectral decomposition of the LBO and is given by:

 HX(t,x,x′)=∑ieλitϕi(x)ϕi(x′) (13)

For every pair of pointwise descriptors , we can compute the corresponding bilateral operators that combine the descriptor similarity measure of 10 with an intrinsic notion of similarity encoded by the heat kernel:

 O(i)X(x,x′) = O(i)Y(y,y′) = (14)

The bilateral operators of equation 14 are conceptually related to a promonient filtering technique used for the problem of image de-noising. Image de-noising is the problem of recovering an image signal from its noisy estimate. As in the case of any noise removal problem, the common intuition is to employ some form of smoothing to the image in which the value of the image function at each pixel is a weighted average of its neighbors. The classical signal processing view of smoothing employs a gaussian weighted averaging of the the image as in equation 15.

 ˆI(pi)∝∑j∈Nie−||pi−pj||22σ2I(pj) (15)

Here encodes the 2D spatial location of pixel , and refers to the pixel indices in the spatial neighborhood of the pixel. is the estimate of the de-noised image. The operation of Equation 15 is equivalent to a heat diffusion of on the standard 2D euclidean plane with the noisy image as the initial condition. This is validated by the fact that the heat kernel in planar domain is the standard gaussian kernel:

 H2D(t,pi,pj)=14πte−||pi−pj||2/4t (16)

However, an important property desirable from such smoothing operators is edge preservation. See figure 1. A gaussian smoothing of equation 15 will smooth away essential details of the image like dominant edges and textures, since the weights are independent of image intensity. This motivated certain edge preserving filters that allowed for a image-dependent heat diffusion. The beltrami flow framework [19, 20] and the the bilateral filter  introduced an additional component that weights a pair of pixels by the difference in their image values in addition to geometric closeness ensured by the heat kernel. Therefore the action of the bilateral filter was to average pixels that were close in the geometric and functional domain, leading to an improved image de-noising filter.

 ˆI(pi)∝∑j∈Ni{e−||pi−pj||22σ2.e−|I(pi)−I(pj)|22β2}I(pj) (17)

In this paper, we assert that a similar argument can be extended to the recovery of correspondence maps. Given the knowledge of good quality point-wise descriptors, we assert that a desirable correspondence map must preserve a combination of both geodesic and descriptor closeness. This can be achieved by commuting with the bilateral operators of equation 14.

There is however a minor conceptual difference between the bilateral operators used in image smoothing and the operators of equation 14. An edge preserving kernel for image smoothing demands that the points be spatially close and also have similar image intensities. Hence the resulting bilateral operator is a pointwise multiplication of the kernels (or a Hadamard multiplication of two positive definite matrices). However, this property is appropriate for a de-noising scenario where both similarities need to be strong for the final weight to be significant.

In contrast, the bilateral operator proposed in this paper suggests an additive combination of geodesic and descriptor similarity. Hence either of the similarities need to be strong for the final kernel value to be significant. The constraint of commuting with such an operator leads to the following observation. Pairs of points on the source shape that are similar in a geodesic sense (hence a higher value for the heat kernel) or by way of having identical descriptor values must correspond distinctly to similar pairs on the target. The optimal correspondence ignores only those pairs that are neither geodesically close nor have similar descriptor values. Intuitively, this policy works favourably when the shapes are not strictly related by an isometry, since there exist some pairs of points that are relatively far apart on the surface but worthy of good matching. The heat kernel may not weight such pairs highly. However, such pairs can be rightly deemed close by an informative pointwise descriptor and this relationship is leveraged by the pairwise bilateral operator. Hence commuting with the bilateral operator enables a more informative utilization of each descriptor leading to a better correspondence quality for a smaller budget of point-wise descriptors.

Figure 2 provides a visualization on the action of the kernel operator of equation 10. The eigenfunctions show a harmonic progression in the descriptor space, that is, the shape is harmonically clustered into regions of similar descriptor values. This is in contrast to the global manifold harmonics that are the eigenfunctions of the heat kernel. Therefore, given that we have knowledge of a pair of corresponding descriptors, we require that, not only must the correspondence map preserve the value of the descriptor pointwise, but also the layered pairwise relationship between every pair of points as encoded by the descriptor centric kernel of Equation 10. This philosophy works best when we combine it with a geometric notion of similarity encoded in the heat kernel.

### 3.1 Numerical Computation

For each descriptor, an explicit construction of the bilateral operators demands the population of a matrix using equation 10. However, for a functional map estimation, we only need the spectral equivalents of the operators for its use in the optimization 6. These can be computed as follows:

 ˆO(i)X = ⎡⎢ ⎢⎣eλ1t⋱eλkt⎤⎥ ⎥⎦+ΦTKf(i)XΦ ˆO(i)Y = ⎡⎢ ⎢⎣eμ1t⋱eμkt⎤⎥ ⎥⎦+ΨTKf(i)YΨ (18)

For spectral projection of the descriptor kernels and , we use the nystrom extension for approximating large positive definite matrices . In particular, the Nystrom approximation requires only a sparse set of columns of any kernel matrix. Let be such a column stacked matrix of the columns of . Let be the smaller intersecting submatrix of . Then we have the following expressions:

 Kf(i)X ≈ RR0−1RT (19) ΦTKf(i)XΦ ≈ (ΦTR)R0−1(ΦTR)T (20)

and a similar expression may be obtained for the kernel matrices for shape . We use the farthest point sampling strategy  for constructing the column matrix . By evaluating the kernel functions 10 for only a sparse subsampled set of points and use the approximations of equation 20 we arrive at the spectral coefficients for the pairwise bilateral operators to be used in a functional map pipeline.

## 5 Experiments

The general philosophy of our experiments is to compare between various functional map algorithms by imputing all of them with the

same set of descriptors and using the same linear optimizer with identical coefficients for the pairwise and pointwise constraints. We compare with three algorithms: the original functional maps framework  without any descriptor based regularizer, the diagonal operator proposed in  and kernel functional maps . All of these algorithms have the same objective of maximizing the information from a fixed budget of pointwise descriptors.

First, we conduct a synthetic experiment to assess the variation in correspondence quality as a function of number of perfect input pointwise descriptors over 25 shape pairs of FAUST 

. The objective of this experiment is to assess which functional map algorithm (or constraint) maximally exploits the information from a fixed budget of high quality descriptors. Most recent and arguably state-of-the-art algorithms on shape correspondence are tailored towards learning optimal descriptors with properties of localization, sensitivity and invariance. Recent results from deep-learning frameworks have demonstrated that such descriptors can be learned from examples

[14, 6, 13, 10]. What we intend to answer with the first experiment is the question, Given that we have high-quality pointwise descriptors, which functional map algorithm achieves the best correspondence quality? Figure 3: Correspondence quality vs number of descriptors for various algorithms.

We compute descriptors by constructing gaussian kernels over ground-truth corresponding points (see Figure 2 (a)), and progressively increase the number of such descriptors as the input to the different algorithms.

 f(i)X=e−d(xi,x)2σ2 (21)

What we expect to see is a decay in the average correspondence geodesic error as the number of such descriptors increase for each algorithm. We use 60 eigenbasis functions for all shapes algorithms. The parameters for all algorithms we chosen as per the choices made in the respective papers. For the bilateral kernel we chose and and for all experiments. Figure 3 shows the results. We see that for a few descriptors, the bilateral filter shows a much lower average correspondence error for the same number of input descriptors. However, all algorithms eventually converge similarly when the number of descriptors increase. Table 1 visualizes the correspondence for the first few landmarks of the plot.

However, general descriptors are far from perfect. This means that there will be a problem of noise leading to inaccurate correspondence. Therefore, in such scenarios, it is not surprising to expect that increasing the number of descriptors may not necessarily lead to an improved correspondence. Moreover the existence of outliers may even deteriorate correspondence quality.

Figure 6 shows the results of our approach compared to ,, on SCAPE and FAUST datasets. We evaluate each algorithm on randomly 100 chosen pairs, for each shape we used 100 discrete laplace-beltarami eigenfunctions as the functional basis. The results of figures 6 show that the bilateral kernel is very potent at lower number of descriptors, and compares favourably with other algorithms for larger number of descriptors, similar to the results in of figure 3. Figure 5 shows comparison of qualitative results of point to point mapping from the source to a target shape using 10 wave kernel signature/map  descriptors. The qualitative results are inline with the quantitative ones, see figure 6.

## 6 Conclusion

In this paper, we propose a novel operator constructed from a pointwise descriptor. We introduce the bilateral operator that combines a descriptor based similarity with the intrinsic geometry of the surface encoded in the heat kernel. By enforcing commutativity constraints with this operator, we demonstrate that we can achieve a better correspondence quality, given a fixed budget of pointwise descriptors. Figure 4: A source shape Table 1: Visualizing correspondence for varying number of descriptors for different algorithms. Functional Map: , Diagonal refers to  and Kernel Functional Map Figure 5: Example of point to point mapping of smooth function on the source (left) to the target, obtained by different functional maps algorithms using 10 wave kernel descriptors. (a) Faust, 10 descriptors

## References

•  Y. Aflalo, H. Brezis, and R. Kimmel. On the optimality of shape and data representation in the spectral domain. SIAM Journal on Imaging Sciences, 8(2):1141–1160, 2015.
•  Y. Aflalo, A. Dubrovina, and R. Kimmel. Spectral generalized multi-dimensional scaling.

International Journal of Computer Vision

, 118(3):380–392, 2016.
•  D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. Scape: shape completion and animation of people. In ACM transactions on graphics (TOG), volume 24, pages 408–416. ACM, 2005.
•  M. Aubry, U. Schlickewei, and D. Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In 2011 IEEE international conference on computer vision workshops (ICCV workshops), pages 1626–1633. IEEE, 2011.
•  F. Bogo, J. Romero, M. Loper, and M. J. Black. Faust: Dataset and evaluation for 3d mesh registration. In

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

, pages 3794–3801, 2014.
•  D. Boscaini, J. Masci, E. Rodolà, and M. Bronstein.

Learning shape correspondence with anisotropic convolutional neural networks.

In Advances in Neural Information Processing Systems, pages 3189–3197, 2016.
•  A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proceedings of the National Academy of Sciences, 103(5):1168–1172, 2006.
•  A. Dubrovina and R. Kimmel. Matching shapes by eigendecomposition of the laplace-beltrami operator. In Proc. 3DPVT, volume 2, page 2, 2010.
•  A. Dubrovina and R. Kimmel. Approximately isometric shape correspondence by matching pointwise spectral features and global geodesic structures. Advances in Adaptive Data Analysis, 3(01n02):203–228, 2011.
•  O. Halimi, O. Litany, E. Rodola, A. M. Bronstein, and R. Kimmel. Unsupervised learning of dense shape correspondence. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
•  D. S. Hochbaum and D. B. Shmoys.

A best possible heuristic for the k-center problem.

Mathematics of operations research, 10(2):180–184, 1985.
•  I. Kokkinos, M. M. Bronstein, R. Litman, and A. M. Bronstein. Intrinsic shape context descriptors for deformable shapes. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 159–166. IEEE, 2012.
•  O. Litany, T. Remez, E. Rodolà, A. Bronstein, and M. Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the IEEE International Conference on Computer Vision, pages 5659–5667, 2017.
•  R. Litman and A. M. Bronstein. Learning spectral descriptors for deformable shape correspondence. IEEE transactions on pattern analysis and machine intelligence, 36(1):171–180, 2013.
•  R. Litman, A. M. Bronstein, and M. M. Bronstein. Stable volumetric features in deformable shapes. Computers & Graphics, 36(5):569–576, 2012.
•  D. Nogneng and M. Ovsjanikov. Informative descriptor preservation via commutativity for shape matching. In Computer Graphics Forum, volume 36, pages 259–267. Wiley Online Library, 2017.
•  M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (TOG), 31(4):30, 2012.
•  M. Ovsjanikov, Q. Mérigot, F. Mémoli, and L. Guibas. One point isometric matching with the heat kernel. In Computer Graphics Forum, volume 29, pages 1555–1564. Wiley Online Library, 2010.
•  N. Sochen, R. Kimmel, and A. M. Bruckstein. Diffusions and confusions in signal and image processing. Journal of Mathematical Imaging and Vision, 14(3):195–209, 2001.
•  A. Spira, R. Kimmel, and N. Sochen. A short-time beltrami kernel for smoothing images and manifolds. IEEE Transactions on Image Processing, 16(6):1628–1636, 2007.
•  J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383–1392. Wiley Online Library, 2009.
•  C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Iccv, volume 98, page 2, 1998.
•  F. Tombari, S. Salti, and L. Di Stefano. Unique signatures of histograms for local surface description. In European conference on computer vision, pages 356–369. Springer, 2010.
•  M. Vestner, Z. Lähner, A. Boyarski, O. Litany, R. Slossberg, T. Remez, E. Rodola, A. Bronstein, M. Bronstein, R. Kimmel, et al. Efficient deformable shape correspondence via kernel matching. In 2017 International Conference on 3D Vision (3DV), pages 517–526. IEEE, 2017.
•  L. Wang, A. Gehre, M. M. Bronstein, and J. Solomon. Kernel functional maps. In Computer Graphics Forum, volume 37, pages 27–36. Wiley Online Library, 2018.
•  C. Williams and M. Seeger. Using the nystroem method to speed up kernel machines. Advances in Neural Information Processing Systems 13, 2001.