In last decade, the amount of geometric data available in the public domain, such as Google 3D Warehouse, has grown dramatically and created the demand for shape search and retrieval algorithms capable of finding similar shapes in the same way a search engine responds to text queries. However, while text search methods are sufficiently developed to be ubiquitously used, the search and retrieval of 3D shapes remains a challenging problem. Shape retrieval based on text metadata, like annotations and tags added by the users, is often incapable of providing relevance level required for a reasonable user experience (see Figure 1).
Content-based shape retrieval using the shape itself as a query and based on the comparison of geometric and topological properties of shapes is complicated by the fact that many 3D objects manifest rich variability, and shape retrieval must often be invariant under different classes of transformations. A particularly challenging setting is the case of non-rigid shapes, including a wide range of transformations such as bending and articulated motion, rotation and translation, scaling, non-rigid deformation, and topological changes. The main challenge in shape retrieval algorithms is computing a shape descriptor, that would be unique for each shape, simple to compute and store, and invariant under different type of transformations. Shape similarity is determined by comparing the shape descriptors.
, geometric moments[14, 30], or spectra , whereas the latter describe the local behavior of the shape in a small patch. Popular examples of local descriptors include spin images , shape contexts , integral volume descriptors  and radius-normal histograms . Using the bag of features paradigm common in image analysis [25, 10], a global shape descriptor counting the occurrence of local descriptors in some vocabulary can be computed .
Recently, there has been an increased interest in the use of diffusion geometry [11, 16] for constructing invariant shape descriptors. Diffusion geometry is closely related to heat propagation properties of shapes and allows obtaining global descriptors, such as distance distributions [24, 19, 8] and Laplace-Beltrami spectral signatures , as well local descriptors such as heat kernel signatures [26, 9]. In particular, heat kernel signatures  showed very promising results in large-scale shape retrieval applications .
One limitation of these methods is that, so far, only geometric information has been considered. However, the abundance of textured models in computer graphics and modeling applications, as well as the advance in 3D shape acquisition [35, 36] allowing to obtain textured 3D shapes of even moving objects, bring forth the need for descriptors also taking into consideration photometric information. Photometric information plays an important role in a variety of shape analysis applications, such as shape matching and correspondence [28, 33]. Considering 2D views of the 3D shape [32, 20], standard feature detectors and descriptors used in image analysis such as SIFT  can be employed. More recently, Zaharescu et al.  proposed a geometric SIFT-like descriptor for textured shapes, defined directly on the surface.
Main contribution. In this paper, we extend the diffusion geometry framework to include photometric information in addition to its geometric counterpart. This way, we incorporate important photometric properties on one hand, while exploiting a principled and theoretically established approach on the other. The main idea is to define a diffusion process that takes into consideration not only the geometry but also the texture of the shape. This is achieved by considering the shape as a manifold in a higher dimensional combined geometric-photometric embedding space, similarly to methods in image processing applications [15, 17]. As a result, we are able to construct local descriptors (heat kernel signatures) and global descriptors (diffusion distance distributions). The proposed data fusion can be useful in coping with different challenges of shape analysis where pure geometric and pure photometric methods fail.
Throughout the paper, we assume the shape to be modeled as a two-dimensional compact Riemannian manifold
(possibly with a boundary) equipped with a metric tensor. Fixing a system of local coordinates on , the latter can be expressed as a matrix
, also known as the first fundamental form. The metric tensor allows to express the length of a vectorin the tangent space at a point as , where repeated indices are summed over following Einstein’s convention.
Given a smooth scalar field on the manifold, its gradient is defined as the vector field satisfying for every point and every infinitesimal tangent vector . The metric tensor defines the Laplace-Beltrami operator that satisfies
for any pair of smooth scalar fields ; here denotes integration with respect to the standard area measure on . Such an integral definition is usually known as the Stokes identity. The Laplace-Beltrami operator is positive semi-definite and self-adjoint. Furthermore, it is an intrinsic property of , i.e., it is expressible solely in terms of . In the case when the metric is Euclidean, becomes the standard Laplacian.
The Laplace-Beltrami operator gives rise to the heat equation,
which describes diffusion processes and heat propagation on the manifold. Here, denotes the distribution of heat at time at point . The initial condition to the equation is some heat distribution , and if the manifold has a boundary, appropriate boundary conditions (e.g. Neumann or Dirichlet) must be specified. The solution of (2) with a point initial heat distribution is called the heat kernel and denoted here by . Using a signal processing analogy, can be thought of as the “impulse response” of the heat equation.
By the spectral decomposition theorem, the heat kernel can be represented as 
are the eigenvalues and
the corresponding eigenfunctions of the Laplace-Beltrami operator (i.e., solutions to). The value of the heat kernel
can be interpreted as the transition probability density of a random walk of lengthfrom the point to the point . This allows to construct a family of intrinsic metrics known as diffusion metrics,111Note that here the term metric is understood in the sense of metric geometry rather than the Riemannian inner product. To avoid confusion, we refer to the latter as to metric tensor throughout the paper.
These metrics have an inherent multi-scale structure and measure the “connectivity rate” of the two points by paths of length . We will collectively refer to quantities expressed in terms of the heat kernel or diffusion metrics as to diffusion geometry. Since the Laplace-Beltrami operator is intrinsic, the diffusion geometry it induces is invariant under isometric deformations of (incongruent embeddings of into ).
3 Fusion of geometric and photometric data
Let us further assume that the Riemannian manifold is a submanifold of some manifold () with the Riemannian metric tensor , embedded by means of a diffeomorphism . A Riemannian metric tensor on induced by the embedding is the pullback metric for , where is the differential of . In coordinate notation, the pullback metric is expressed as , where the indices denote the embedding coordinates.
Here, we use the structure of to model joint geometric and photometric information. Such an approach has been successfully used in image processing . When considering shapes as geometric object only, we define and to be the Euclidean metric. In this case, acts as a parametrization of and the pullback metric becomes simply . In the case considered in this paper, the shape is endowed with photometric information given in the form of a field , where denotes some colorspace (e.g., RGB or Lab). This photometric information can be modeled by defining and an embedding . The embedding coordinates corresponding to geometric information are as previously and the embedding coordinate corresponding to photometric information are given by , where is a scaling constant. Simplifying further, we assume to have a Euclidean structure (for example, the Lab colorspace has a natural Euclidean metric). The metric in this case boils down to , which hereinafter we shall denote by .
The Laplace-Beltrami operator associated with such a metric gives rise to diffusion geometry that combines photometric and geometric information (Figure 2).
Invariance. It is important to mention that the joint metric tensor and the diffusion geometry it induces have inherent ambiguities. Let us denote by and the respective groups of transformation that leave the geometric and the photometric components of the shape unchanged. We will refer to such transformations as geometric and photometric isometries. The diffusion metric induced by is invariant the joint isometry group . Ideally, we would like to hold. In practice, is bigger: while every composition of a geometric isometry with a photometric isometry is a joint isometry, there exist some joint isometries which cannot be obtained as a composition of geometric and photometric isometries. An example of such transformations is uniform scaling of combined with compensating scaling of . The ambiguity stems from the fact that is bigger compared to . Experimental results show that no realistic geometric and photometric transformations lie in , however, a formal characterization of the isometry group is an important theoretical question for future research.
4 Numerical implementation
Let denote the discrete samples of the shape, and be the corresponding embedding coordinates (three-dimensional in the case we consider only geometry, or six-dimensional in the case of geometry-photometry fusion). We further assume to be given a triangulation (simplicial complex), consisting of edges and faces where each , and is an edge (here ).
Discrete Laplacian. A function on the discretized manifold is represented as an -dimensional vector . The discrete Laplace-Beltrami operator can be written in the generic form
where are weights, are normalization coefficients, and denotes a local neighborhood of point . Different discretizations of the Laplace-Beltrami operator can be cast into this form by appropriate definition of the above constants. For shapes represented as triangular meshes, a widely-used method is the cotangent scheme, which preserves many important properties of the continuous Laplace-Beltrami operator, such as positive semi-definiteness, symmetry, and locality . Yet, in general, the cotangent scheme does not converge to the continuous Laplace-Beltrami operator, in the sense that the solution of the discrete eigenproblem does not converge to the continuous one (pointwise convergence exists if the triangulation and sampling satisfy certain conditions ).
Belkin et al.  proposed a discretization which is convergent without the restrictions on “good” triangulation required by the cotangent scheme. In this scheme, is chosen to be the entire sampling , , and , where is a parameter. In the case of a Euclidean colorspace, can be written explicitly as
where , which resembles the weights used in the bilateral filter . Experimental results also show that this operator produces accurate approximation of the Laplace-Beltrami operator under various conditions, such as noisy data input and different sampling [27, 5].
Heat kernel computation. In matrix notation, equation (5) can be written as , where and . The eigenvalue problem is equivalent to the generalized symmetric eigenvalue problem , where is the diagonal matrix of the first eigenvalues, and
is the matrix of the eigenvectors stacked as columns. Since typicallyis sparse, this problem can be efficiently solved numerically.
Heat kernels can be approximated by taking the first largest eigenvalues and the corresponding eigenfunctions in (3). Since the coefficients in the expansion of decay as , typically a few eigenvalues ( in the range of to ) are required.
5 Results and applications
In this section, we show the application of the proposed framework to retrieval of textured shapes. We compare two approaches: bags of local features and distributions of diffusion distances.
5.1 Bags of local features
ShapeGoogle framework. Sun et al.  proposed using the heat propagation properties as a local descriptor of the manifold. The diagonal of the heat kernel,
referred to as the heat kernel signature (HKS), captures the local properties of at point and scale . The descriptor is computed at each point as a vector of the values , where are some time values. Such a descriptor is deformation-invariant, easy to compute, and provably informative .
Ovsjanikov et al.  employed the HKS local descriptor for large-scale shape retrieval using the bags of features paradigm . In this approach, the shape is considered as a collection of “geometric words” from a fixed “vocabulary” and is described by the distribution of such words, also referred to as a bag of features or BoF. The vocabulary is constructed offline by clustering the HKS descriptor space. Then, for each point on the shape, the HKS is replaced by the nearest vocabulary word by means of vector quantization. Counting the frequency of each word, a BoF is constructed. The similarity of two shapes and is then computed as the distance between the corresponding BoFs, .
Using the proposed approach, we define the color heat kernel signature (cHKS), defined in the same way as HKS with the standard Laplace-Belrami operator replaced by the one resulting from the geometric-photometric embedding. In the following, we show that such descriptors allow achieving superior retrieval performance.
Evaluation methodology. In order to evaluate the proposed method, we used the SHREC 2010 robust large-scale shape retrieval benchmark methodology . The query set consisted of 270 real-world human shapes from 5 classes acquired by a 3D scanner with real geometric transformations and simulated photometric transformations of different types and strengths, totalling in 54 instances per shape (Figure 3). Geometric transformations were divided into isometry+topology (real articulations and topological changes due to acquisition imperfections), and partiality (occlusions and addition of clutter such as the red ball in Figure 3). Photometric transformations included contrast (increase and decrease by scaling of the channel), brightness (brighten and darken by shift of the channel), hue (shift in the channel), saturation (saturation and desaturation by scaling of the channels), and color noise (additive Gaussian noise in all channels). Mixed transformations included isometry+topology transformations in combination with two randomly selected photometric transformations. In each class, the transformation appeared in five different versions numbered 1–5 corresponding to the transformation strength levels. One shape of each of the five classes was added to the queried corpus in addition to other 75 shapes used as clutter (Figure 4).
Retrieval was performed by matching 270 transformed queries to the 75 null shapes. Each query had exactly one correct corresponding null shape in the dataset. Performance was evaluated using the precision-recall characteristic. Precision is defined as the percentage of relevant shapes in the first top-ranked retrieved shapes. Mean average precision (mAP), defined as , where is the relevance of a given rank, was used as a single measure of performance. Intuitively, mAP is interpreted as the area below the precision-recall curve. Ideal retrieval performance results in first relevant match with mAP=100%. Performance results were broken down according to transformation class and strength.
Methods. In additional to the proposed approach, we compared purely geometric, purely photometric, and joint photometric-geometric descriptors. As a purely geometric descriptor, we used bags of features based on HKS according to ; purely photometric shape descriptor was a color histogram. As joint photometric-geometric descriptors, we used bags of features computed with the MeshHOG  and the proposed color HKS (cHKS).
For the computation of the bag of features descriptors, we used the Shape Google framework with most of the settings as proposed in . More specifically, HKS were computed at six scales (, and
). Soft vector quantization was applied with variance taken as twice the median of all distances between cluster centers. Approximate nearest neighbor method was used for vector quantization. The Laplace-Beltrami operator discretization was computed using the Mesh-Laplace scheme  with scale parameter . Heat kernels were approximated using the first eigenpairs of the discrete Laplacian. The MeshHOG descriptor was computed at prominent feature points (typically 100-2000 per shape), detected using the MeshDOG detector . The vocabulary size in all the cases was set to .
In cHKS, in order to avoid the choice of an arbitrary value , we used a set of three different weights () to compute the cHKS and the corresponding BoFs. The distance between two shapes was computed as the sum of the distances between the corresponding BoFs for each , weighted by , and 1 in case of , .
Results. Tables 1–4 summarize the results of our experiments. Geometry only descriptor (HKS)  is invariant to photometric transformations, but is somewhat sensitive to topological noise and missing parts (Table 1). On the other hand, the color-only descriptor works well only for geometric transformations that do not change the shape color. Photometric transformations, however, make such a descriptor almost useless (Table 2). MeshHOG is almost invariant to photometric transformations being based on texture gradients, but is sensitive to color noise (Table 3). The fusion of the geometric and photometric data using our approach (Table 4) achieves nearly perfect retrieval for mixed and photometric transformations and outperforms other approaches. Figure 5 visualizes a few examples of the retrieved shapes ordered by relevance, which is inversely proportional to the distance from the query shape.
5.2 Shape distributions
Spectral shape distances. Recent works [24, 19] showed that global shape descriptors can be constructed considering distributions of intrinsic distances. Given some intrinsic distance metric , its cumulative distribution is computed as
where denotes an indicator function. Given two shapes and with the corresponding distance metrics , the similarity (referred to as spectral distance) is computed as a distance between the corresponding distributions and .
Using the proposed framework, we construct diffusion distances according to (4), where the standard Laplace-Beltrami operator is again replaced by the one associated with the geometric-photometric embedding. Such distances account for photometric information, and, as we show in the following, show superior performance.
Methods. Using the same benchmark as above, we compared shape retrieval approaches that use distance distribution as shape descriptors. Two methods were compared: pure geometric and joint geometric-photometric distances. In the former, we used average of diffusion distances
computed at two scales, . In the latter, the distances were also computed at multiple scales of the photometric component,
The values were used. For the computation of distributions, the shapes were subsampled at points using the farthest point sampling algorithm.
Results. Tables LABEL:tab:regular:diff–LABEL:tab:mltsc-sum summarize the results. Both descriptors appear insensitive to photometric transformations. The joint distance has superior performance in pure geometric and mixed transformations. We conclude that the use of non-zero weight for the color component adds discriminativity to the distance distribution descriptor, while being still robust under photometric transformations.
In this paper, we explored a way to fuse geometric and photometric information in the construction of shape descriptors. Our approach is based on heat propagation on a manifold embedded into a combined geometry-color space. Such diffusion processes capture both geometric and photometric information and give rise to local and global diffusion geometry (heat kernels and diffusion distances), which can be used as informative shape descriptors. We showed experimentally that the proposed descriptors outperform other geometry-only and photometry-only descriptors, as well as state-of-the-art joint geometric-photometric descriptors. In the future, it would be important to formally characterize the isometry group induced by the joint metric in order to understand the invariant properties of the proposed diffusion geometry, and possibly design application-specific invariant descriptors.
-  J. Amores, N. Sebe, and P. Radeva. Context-based object-class recognition and retrieval by generalized correlograms. Trans. PAMI, 29(10):1818–1833, 2007.
-  S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Y. Wu. An optimal algorithm for approximate nearest neighbor searching. J. ACM, 45:891–923, 1998.
-  J. Assfalg, M. Bertini, A.D. Bimbo, and P. Pala. Content-based retrieval of 3-d objects using spin image signatures. Multimedia, IEEE Transactions on, 9(3):589 –599, apr. 2007.
-  M. Belkin, J. Sun, and Y. Wang. Constructing Laplace operator from point clouds in Rd. In Proc. Symp. Discrete Algorithms, pages 1031–1040, 2009.
-  M. Belkin, J. Sun, and Y. Wang. Discrete Laplace operator on meshed surfaces. In Proc. Symp. Computational Geometry, pages 278–287, 2009.
-  A. M. Bronstein, M. M. Bronstein, U. Castellani, B. Falcidieno, A. Fusiello, A. Godil, L. J. Guibas, I. Kokkinos, Z. Lian, M. Ovsjanikov, G. Patané, M. Spagnuolo, and R. Toldo. Shrec 2010: robust large-scale shape retrieval benchmark. In Proc. 3DOR, 2010.
A. M. Bronstein, M. M. Bronstein, M. Ovsjanikov, and L. J. Guibas.
Shape google: a computer vision approach to invariant shape retrieval.In Proc. NORDIA, 2009.
-  M. M. Bronstein and A. M. Bronstein. Shape recognition with spectral distances. Trans. PAMI, 2010. to appear.
-  M. M. Bronstein and I. Kokkinos. Scale-invariant heat kernel signatures for non-rigid shape recognition. In Proc. CVPR, 2010.
-  O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman. Total recall: Automatic query expansion with a generative feature model for object retrieval. In Proc. ICCV, 2007.
-  R. R. Coifman and S. Lafon. Diffusion maps. Applied and Computational Harmonic Analysis, 21:5–30, July 2006.
-  N. Gelfand, N. J. Mitra, L. J. Guibas, and H. Pottmann. Robust global registration. In Proc. SGP, 2005.
-  P. W. Jones, M. Maggioni, and R. Schul. Manifold parametrizations by eigenfunctions of the Laplacian and heat kernels. PNAS, 105(6):1803, 2008.
-  M. Kazhdan, T. Funkhouser, and S. Rusinkiewicz. Rotation invariant spherical harmonic representation of 3D shape descriptors. In Proc. SGP, pages 156–164, 2003.
-  R. Kimmel, R. Malladi, and N. Sochen. Images as embedded maps and minimal surfaces: movies, color, texture, and volumetric medical images. IJCV, 39(2):111–129, 2000.
-  B. Lévy. Laplace-Beltrami eigenfunctions towards an algorithm that “understands” geometry. In Proc. Shape Modeling and Applications, 2006.
-  H. Ling and D. W. Jacobs. Deformation invariant image matching. In In ICCV, pages 1466–1473, 2005.
-  D. Lowe. Distinctive image features from scale-invariant keypoint. IJCV, 2004.
-  M. Mahmoudi and G. Sapiro. Three-dimensional point cloud recognition via distributions of geometric distances. Graphical Models, 71(1):22–31, January 2009.
-  R. Ohbuchi, K. Osada, T. Furuya, and T. Banno. Salient local visual features for shape-based 3d model retrieval. pages 93 –102, jun. 2008.
-  R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin. Shape distributions. TOG, 21(4):807–832, 2002.
-  X. Pan, Y. Zhang, S. Zhang, and X. Ye. Radius-normal histogram and hybrid strategy for 3d shape retrieval. pages 372 – 377, jun. 2005.
-  M. Reuter, F.-E. Wolter, and N. Peinecke. Laplace-spectra as fingerprints for shape matching. In Proc. ACM Symp. Solid and Physical Modeling, pages 101–106, 2005.
-  R. M. Rustamov. Laplace-Beltrami eigenfunctions for deformation invariant shape representation. In Proc. SGP, pages 225–233, 2007.
-  J. Sivic and A. Zisserman. Video google: A text retrieval approach to object matching in videos. In Proc. CVPR, 2003.
-  J. Sun, M. Ovsjanikov, and L. J. Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Proc. SGP, 2009.
-  K. Thangudu. Practicality of Laplace operator, 2009.
-  N. Thorstensen and R. Keriven. Non-rigid shape matching using geometry and photometry. In Proc. CVPR, 2009.
-  C. Tomasi and R. Manduchi. Bilateral fitering for gray and color images. In Proc. ICCV, pages 839–846, 1998.
-  D. V. Vranic, D. Saupe, and J. Richter. Tools for 3D-object retrieval: Karhunen-Loeve transform and spherical harmonics. In Proc. Workshop Multimedia Signal Processing, pages 293–298, 2001.
-  M. Wardetzky, S. Mathur, F. Kälberer, and E. Grinspun. Discrete Laplace operators: no free lunch. In Conf. Computer Graphics and Interactive Techniques, 2008.
-  C. Wu, B. Clipp, X. Li, J.-M. Frahm, and M. Pollefeys. 3d model matching with viewpoint-invariant patches (vip). pages 1 –8, jun. 2008.
-  J.V. Wyngaerd. Combining texture and shape for automatic crude patch registration. pages 179 – 186, oct. 2003.
-  G. Xu. Convergence of discrete Laplace-Beltrami operators over surfaces. Technical report, Institute of Computational Mathematics and Scientific/Engineering Computing, China, 2004.
K.-J. Yoon, E. Prados, and P. Sturm.
Joint estimation of shape and reflectance using multiple images with known illumination conditions, 2010.
-  A. Zaharescu, E. Boyer, and R. P. Horaud. Transformesh: a topology-adaptive mesh-based approach to surface evolution, November 2007.
-  A. Zaharescu, E. Boyer, K. Varanasi, and R Horaud. Surface feature detection and description with applications to mesh matching. In Proc. CVPR, 2009.