Conformal Surface Morphing with Applications on Facial Expressions

04/01/2015
by   Mei-Heng Yueh, et al.
0

Morphing is the process of changing one figure into another. Some numerical methods of 3D surface morphing by deformable modeling and conformal mapping are shown in this study. It is well known that there exists a unique Riemann conformal mapping from a simply connected surface into a unit disk by the Riemann mapping theorem. The dilation and relative orientations of the 3D surfaces can be linked through the Möbius transformation due to the conformal characteristic of the Riemann mapping. On the other hand, a 3D surface deformable model can be built via various approaches such as mutual parameterization from direct interpolation or surface matching using landmarks. In this paper, we take the advantage of the unique representation of 3D surfaces by the mean curvatures and the conformal factors associated with the Riemann mapping. By registering the landmarks on the conformal parametric domains, the correspondence of the mean curvatures and the conformal factors for each surfaces can be obtained. As a result, we can construct the 3D deformation field from the surface reconstruction algorithm proposed by Gu and Yau. Furthermore, by composition of the Möbius transformation and the 3D deformation field, the morphing sequence can be generated from the mean curvatures and the conformal factors on a unified mesh structure by using the cubic spline homotopy. Several numerical experiments of the face morphing are presented to demonstrate the robustness of our approach.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 5

page 7

09/10/2020

Efficient conformal parameterization of multiply-connected surfaces using quasi-conformal theory

Conformal mapping, a classical topic in complex analysis and differentia...
08/30/2020

A diffusion-driven Characteristic Mapping method for particle management

We present a novel particle management method using the Characteristic M...
01/21/2022

Combined parameterization of material distribution and surface mesh for stiffener layout optimization of complex surfaces

Stiffener layout optimization of complex surfaces is fulfilled within th...
06/01/2016

Multiview Rectification of Folded Documents

Digitally unwrapping images of paper sheets is crucial for accurate docu...
08/01/2010

Parametric polynomial minimal surfaces of arbitrary degree

Weierstrass representation is a classical parameterization of minimal su...
08/29/2014

Fast Disk Conformal Parameterization of Simply-connected Open Surfaces

Surface parameterizations have been widely used in computer graphics and...
03/29/2019

Parallelizable global conformal parameterization of simply-connected surfaces via partial welding

Conformal surface parameterization is useful in graphics, imaging and vi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The metamorphosis between two objects is commonly called ”morphing”. It is the process of changing one figure into another. In recent years, image morphing techniques have been widely used in the entertainment industry. Many techniques have been developed to achieve a desired morphing effect. [17]

In 2D image morphing, S.-Y. Lee et al. [12] proposed a technique that generates a -continuous and one-to-one deformation from the two-dimensional positional constraints by using a deformable surface model. The transition behavior can be controlled by assigning the transition curves for selected points on an image. Similarly, A. Gregory et al. [6] proposed a method of geometry morphing, which creates a morph by defining morphing trajectories between the feature pairs and by interpolating them across the merged polyhedron. Attributed to these technologies, the user can achieve a satisfactory visual effect by simply setting the correspondence between graphical features. In particular, for images of human faces, it is possible to pick the feature points of the face automatically. V. Zanella et al. [19] use the so-called ”Active Shape Models” [3] to find the facial features in the 2D images and perform the morphing of face images in frontal view automatically. To make the metamorphosis look realer, the morphing path must be constrained by some physical laws. Researchers, such as Verbeek et al. [16], Y. Bao et al. [1] and L. Younes et al. [2], proposed methods of determining the morphing paths by optimizing an energy functional, which characterizes the intrinsic deformation of the surface away from its rest shape. Althrough 2D image morphing technique has pretty mature, 3D image morphing remains challenges, especially when the virtual real morphing effects are desired. In addition, in order to achieve a satisfactory visual effect, the texture images also need to be computed in the process of visualization.

On the other hand, with the advance of the three-dimensional imaging technology, surface morphing in 3D has become very important. Comparing to the 2D image matching problem, surface matching problem is much more difficult, since the surface matching involves the correspondence in coordinates and the geometric information of images in is far richer than images in 2D. D. DeCarlo et al. [4] proposed a method of generating smooth-looking transformations between pairs of surfaces that may differ in topology by specifying a sparse control mesh on each surface and by associating each face in one control mesh with a corresponding face in the other. Y.-S. Liu et al. [14] proposed a morphing method that works well even when two input 3D triangle meshes have very different shapes by creating consistent meshes for the original source and target models. Schröder et al. [13] proposed a variational approach based on minimizing bending and stretching in which corresponding feature points and line segments are matched. E. Jeong et al. [11] proposed a feature-based morphing technique for two objects equipped with surface light fields by using spherical embedding of meshes. M. Yang et al. [18] proposes a realistic 3D morphing method based on GPU for real-time animation of facial expressions by rendering the real texture on the digital character.

In this study, we propose an efficient way to obtain the desired morphing effects. Our goal is to generate the morphing sequence between human faces in which features of the given surfaces are maintained during the morphing process. Moreover, in order to control digital models and perform digital acting via real agents, we also construct the single mesh of human faces through the correspondence matching. In our approach, we utilize the conformal mapping technique. A Möbius transformation and a deformation from a modified thin-plate model are computed to achieve high accurate feature correspondence. A single mesh based on geodesic paths among feature points is computed very efficiently. With the help provided by this single mesh, the texture information and the geometric information, such as the conformal factor and the mean curvature , of each frame in the morphing sequence can be easily interpolated from the given surfaces through various homotopy methods. Then the 3D images of each intermediate frame can be computed from the reconstruction algorithm proposed by Gu and Yau. Our method guarantees the piecewise smoothness of homotopy path. So, we can avoid the vibration of each frames caused by the unstableness of the 3D camera. In addition, we merely require 10% the number of original frames to simulate the real motion of human facial expressions, so that the size of the information data is significantly reduced but the resolution of the 3D images does not decrease.

In the following, we review the computation of the Riemann conformal mapping in section 2. We present our surface matching technique in section 3 and introduce the way of homotopy in section 4. After that, we show how we reconstruct the morphing sequence in section 6 and demonstrate some morphing results in section 7.

2 Idea for Computing Conformal Mapping

The Riemann conformal mapping plays an important role in the surface matching. In this section, we briefly review the method of computing Riemann conformal mapping.

2.1 Spherical Conformal Mapping

[height=2.5cm]images/arrow_right.png Spherical Conformal

Fig. 1: Spherical conformal mapping

The spherical conformal mapping is first computed by Gu and Yau [8] in 2003. The idea is based on minimizing the harmonic energy through a nonlinear heat diffusion process as following: Suppose the desired map is Let and denote the image of the vertex and the normal at respectively. The normal and tangent components of are defined as

and

respectively. The harmonic map is then computed by minimizing the harmonic energy associated with through the nonlinear heat diffusion process

with the constrain . The equation can be written in the following form,

and can be solved numerically by using the quasi-implicit Euler method (QIEM) [7],

where, is the discrete Laplacian, is a diagonal matrix with

The initial map for the above iterative formula can be obtained from the Gauss map. It is well known that the efficiency of the explicit scheme is usually not satisfactory since the time step is generally very small due to the diffusive nature. The quasi-implicit Euler method [7] proposed by W.-W. Lin et al. is shown to be very robust on solving the nonlinear diffusion equation.

2.2 Riemann Conformal Mapping

The idea for computing Riemann conformal mapping, shown in Figure 2, was first proposed by Gu and Yau [10]. For a given surface with single boundary, we make it into a closed surface by using the double covering technique. Then we compute the spherical conformal mapping of . Finally, we cut out the semi-sphere along the equator and map the unit semi-sphere onto the unit disk conformally by using the stereographic projection.

[height=2cm]images/face_mesh.png             [height=2cm]images/arrow_right.png Riemann Map[height=2cm]images/face_disk.png
[height=0.8cm]images/arrow_down.png Double Covering[height=0.8cm]images/arrow_up.png Stereographic ProjectionMöbius Transform
[height=2cm]images/face_mesh2.png [height=2cm]images/arrow_right.png Spherical Conformal[height=2cm]images/face_sphere.png

Fig. 2: Idea for computing Riemann conformal mapping

2.3 Numerical Results

We demonstrate the robustness of the quasi-implicit Euler method by computing the Riemann conformal mappings of human facial expressions. Two different facial expressions and the associated conformal mappings are shown in Figure 3. In order to check the conformality of the QIEM, we paste the checkerboard grid on the image of the Riemann conformal mapping , and put it back to the surface by using the inverse of the Riemann conformal mapping . If the mapping is angle-preserving, every angle should be nearly 90 degrees. The histograms of the angle distribution, shown in Figure 3, indicate that the QIEM is very accurate on angle-preserving.

Fig. 3: The results of the Riemann conformal mapping

3 Surface Matching with Minimal Global Error

Surface matching plays a critical role in surface morphing. L. Younes et al. [2] proposed a simple method to interpolate 2D landmark matching by constructing a diffeomorphism between two domains. When it comes to surface matching, it would be much more difficult. However, with the Riemann conformal mappings, we reduce the surface matching problem into the unit disk matching problem. Hence we can apply the similar idea of the 2D landmark matching to the surface matching.

For the surface matching, X. Gu et al. [8] use landmarks to obtain the optimal Möbius transformation as the matching function between the spherical conformal mappings of two brains. When it comes to the matching between two different facial expressions, a slight twist is allowed since the transformation of facial expressions is actually not conformal. In the following, we propose to match the landmarks of each facial expressions by composition of the Möbius transformation and deformation from the plate matching.

We assume that the correspondence of the boundary points of two different human faces and are known. The assumption can be realized by putting markers on the boundary of the human faces during scanning. Suppose landmarks are selected for each facial expression and denoted by

and

Our purpose is to construct a matching function where First, each surface is mapped to the unit disk by the Riemann conformal mappings

We denote by and by , respectively,

[height=2.8cm]images/mshuang_chewing_0001_feature16.png [height=3cm]images/arrow_right.png [height=2.8cm]images/yylin_0366_feature16.png
[height=1cm]images/arrow_down.png [height=1cm]images/arrow_down.png
[height=2.5cm, scale=0.25, tics=10, angle=0]images/mshuang_chewing_0001_disc_feature16.png       [height=3cm]images/arrow_right.png [height=2.5cm, scale=0.25, tics=10, angle=0]images/yylin_0366_disc_feature16.png

Fig. 4: The idea of surface matching

Next, a uniform grid points of an checkerboard ranged on , , is lay on . In the thin-plate model, the deformation field is approximated by the span of the Green functions of the bending operator at each grid point where is the distance between and . Therefore the matching function between the unit disks is defined by with and

for where are unknown coefficients, . To determine these coefficients for matching landmarks on conformal parametric domain, we solve the least square problems

where

and are the conformal factors, resulted from the Riemann conformal mappings and , at , respectively, and

The least square problem can be easily solved by QR method when or solved by Tikhonov regularization [5]

when Then the matching function between and can be obtained by taking Figure 5 shows the result of global matching and Table I shows the comparison between the optimal Möbius transformation (OMT) and the optimal Möbius transformation with the global matching function (OMGMF).

[height=2cm]images/DeformM.png

Fig. 5: The deformation field of where is the optimal Möbius transformation (OMT) and is the global matching function (GMF)

There is a variety ways to measure the distortion of the matching function, such as Euclidean norm on the space, the infinite norm on the unit disk, etc. In order to verify that the matching pairs are mapped to by we define the conformal matching energy

which measures the square of Euclidean distance at each matching pair on the conformal disk. On another point of view, we define the local matching energy

which directly measures the square of Euclidean distance at each matching pair in . In order to ensure that the global error is relatively small, we also compare the global matching energy, which is defined by

According to Table I, the global matching function significantly reduces the matching energies, which indicates that the global matching function works well on the surface matching.

OMT OMGMF
TABLE I: Comparison between the optimal Möbius transformation (OMT) and the optimal Möbius transformation with the global matching function (OMGMF)

4 Surface Morphing

The effect of the traditional image morphing by using the direct interpolation is not satisfactory since there are shadows in the area of wrong correspondence. In 3D morphing, it could be even worse. To improve this phenomenon, D. Smythe proposed the mesh warping technique [17], which partitions the 2D image into several pieces and interpolate them piece by piece. Base on this idea, we compute a geodesic frame on the surfaces in 3D. The frame consists of the discrete surface geodesics that connect preselected feature points. The frame can be refined to better capture characters of various facial expressions. The geodesic frames for the eight facial expressions are shown in Figure 6.

The method of computing discrete surface geodesics on triangular meshes was proposed by D. Martínez et al. [15] which computes the initial path by using Sethian’s fast marching method and correct the path with the path correcting iterations. For a mesh with a large number of vertices, the fast marching method could be very time consuming. To improve the efficiency, we calculate the initial path by

  • construct the frame on the unit disk, where feature points are connected by straight line segments,

  • the initial paths are obtained by taking the inverse conformal map of these line segments.

  • Apply Martínez’s algorithm to obtain the geodesic frame.

The resulted geodesic frame is called the single mesh. In our calculation, the initial paths in the frame mostly converge to the geodesics within 5 steps in the path correcting iterations. The geodesic frame is remapped to partition the conformal parametric domain as shown in Figure 7.

neutral smile sad pout
bitter smile pain wry ferocious
Fig. 6: The geodesics on a human face with different facial expressions
neutral smile sad pout
bitter smile pain wry ferocious
Fig. 7: The partition mesh of the unit disk for different facial expressions

5 Surface Registration via the Geodesic Frame of the Surface

To build an one-to-one surface registration, we rely on the aforementioned partition mesh . Let be the partition mesh on . The partition mesh on is determined by the where is the matching function discussed in section 3. To introduce our registration map, first let us introduce some notations. The partition mesh of the unit disk is denoted by , here

and

are the set of vertices and the set of triangles in where is the coordinate of the -th vertex and is the indices of the vertices of the -th triangle, and denote the number of vertices in and the number of triangles in . Obviously, the closed region determined by can be easily represented by

where is the barycentric coordinate of with respect to the vertices , and . A simple piecewise linear registration map can be easily constructed by the affine mapping

if , between each pair of triangles and , where

Recall is a unique representation of So, the surface registration can be realized through

Similarly, the texture images and of the surface and , respectively, can be registrated by

In the following, we introduce how we utilize the above surface registration method to generate the morphing sequence through the cubic spline homotopy of the mean curvatures and the conformal factors. Suppose 3D images of facial expressions are captured a time . Using the above surface registration method, the registration maps can be easily computed. Using these registration maps, a morphing path , and , can be created, here denotes the location where a point is morphed at time . Since is a unique representation of a surface, the morphing path can also be uniquely determined by the evolution of the conformal factor and the mean curvature. Here, we employee a piecewise cubic spline homotopy to interpolate

A sketch in Figure 8 illustrates this idea. Let denote the piecewise cubic spline function with given data . The conformal factor and the mean curvature at can now be evaluated by

Similarly, suppose the texture images of the surface , are given. The texture image associated with the surface along the morphing path can be computed by the cubic spline homotopy

Finally, by reconstructing the 3D surfaces from their representation and applying the computed texture image to , the morphing sequence between and can be obtained for any period from to . In the following section, we shall introduce the surface reconstrution algorithm in detail.

[height=2.5cm]images/homotopy.png

Fig. 8: The cubic spline homotopy of the mean curvature and conformal factor of a vertex

6 Laplace-Beltrami Surface Reconstruction

In the previous section, we have obtained the mean curvatures and the conformal factors at each time period , . Gu and Yau proposed that any surface in 3D Euclidean space can be determined by its conformal factor and mean curvature uniquely up to rigid motions [9]. We could generate the morphing sequence of surfaces by reconstructing the unique surface via , .

In the following, we introduce how we utilize the mean curvatures and the conformal factors to reconstruct surfaces by solving the Laplace-Beltrami equations. Applying Gu and Yau’s method, for the given mean curvature and the conformal factor , we reconstruct the unique 3D surface by solving the Laplace-Beltrami equations

where

and is the normal of the surface . The detail algorithm for solving the Laplace-Beltrami equations can be seen in Algorithm 1.

0:  the mean curvature and the conformal factor of the surface and the boundary od the surface .
0:  the surface .
1:  Set the initial surface normal
2:  repeat
3:     Solve the boundary value problem
where boundary is known.
4:     Update the surface normal
5:  until convergence
Algorithm 1 Laplace-Beltrami Surface Reconstruction

We reconstruct 7 frames from the video at time respectively. The surface captured from the video at time is denoted by and the reconstructed surface is denoted by . In order to verify that the reconstructed surface is approximately the real surface, we compute the difference between the real surface and the reconstructed surface in sense

and in sense

respectively. Table II shows the reconstruction error in both norm and norm.

t
TABLE II: Reconstruction error between the real surface and the reconstructed surface

7 Numerical Results of Surface Morphing in 3D

In the following, we show some surface morphing results via merely two images by using the surface morphing technique which we have mentioned above. In each figure, the left most image is the role of initial surface while the right most image is the role of terminal surface and the images in the middle are the morphing sequence between and .

Figure 9 and Figure 10 show the morphing sequence between two different facial expression, respectively. Figure 11 and Figure 12 show the morphing sequence between faces of two different people, respectively. Figure 13 shows the morphing sequence between a human face and a loin’s head.

It is interesting that how much the global matching function improves the morphing sequence. The affine mapping constructed by using the optimal Möbius transformation is denoted by and the affine mapping constructed by using the optimal Möbius transformation with the global matching function is denoted by , . In order to measure the rate of improvement, we reconstruct the surfaces

and

where

and

Then, we compute the surface difference

and

The rate of improvement is defined by

Table III indicates that the global matching function improves approximately of the surface difference.

[height=2cm]images/cwchang_0025.png [height=2cm]images/cwchang_0034.png [height=2cm]images/cwchang_0038.png [height=2cm]images/cwchang_0042.png [height=2cm]images/cwchang_0050.png


Improvement Rate
0.3 42.08%
0.4 45.37%
0.5 54.22%
1.3 55.01%
1.4 51.36%
TABLE III: Comparison between the optimal Möbius transformation and the optimal Möbius transformation with the global matching function
Fig. 9: The morphing sequence of eye blinking
Fig. 10: The morphing sequence of mouth opening
Fig. 11: The morphing sequence from a girl’s face into a boy’s face
Fig. 12: The morphing sequence from a girl’s face into another
Fig. 13: The morphing sequence from Max Planck’s face into a lion’s head

Moreover, extrapolation can also be achieved by using this homotopy technique. The real surface at time is denoted by and the reconstructed surface at time is denoted by where

Table IV shows the surface difference between the real surface and the reconstructed surface in sense

and in sense

respectively, at time and

[height=2cm]images/err_cwchang_0043.png [height=2cm]images/err_cwchang_0044.png [height=2cm]images/err_cwchang_0045.png [height=2cm]images/err_cwchang_0046.png [height=2cm]images/bar.png

1.4
1.5
1.6
1.7
TABLE IV: Extrapolation error between the real surface and the reconstructed surface

8 Conclusion

In this paper, we proposed a 3D surface morphing method between different simply connected surfaces with single boundary in which smooth transient on both geometric characteristics and texture of the surfaces is considered. Similar to the traditional morphing approaches based on boundary representation, a wrap has to be created via feature correspondence and interpolation between shapes based on the wrap is employed to generate the morphing sequence. By taking advantage of the conformal parameterization and the unique surface representation of conformal factor and mean curvature, the wrap can be easily obtained by the composition of deformations from the Möbius transformation and the thin-plate matching function. To mimic the non-isomorphic risk that usually occurs in matching largely deformed surfaces, a single mesh based on geodesic frame is employed. As a result, the correspondence, including geometric information and texture information, of the whole surface can be defined and interpolation among original surface and target surface can be computed by the usual cubic spline homotopy in a disk parametric domain. Finally, the morphing sequence can be generated from the surface reconstruction algorithm in section 6. To make the proposed morphing approach more attractive in real applications, we improve the efficiency in computing the conformal parameterization and geodesic frames. We propose an non-linear iterative surface reconstruction algorithm (Algorithm 1). The surface reconstruction algorithm can be accelerated by using the multigrid method on a uniform mesh by which multi-resolution surfaces can also be obtained. Several morphing effects among different 3D facial expressions are presented to demonstrate the feasibility of the proposed morphing method.

References

  • [1] Y. Bao, X. Guo, and H. Qin. Physically based morphing of point-sampled surfaces. Computer Animation and Virtual Worlds, 16:509–518, 2005.
  • [2] V. Camion and L. Younes. Geodesic interpolating splines.

    Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture notes in Computer Science

    , 2134:513–527, 2001.
  • [3] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Actives shape models - their training and application. Computer Vision and Image Understanding, 61(1), 1995.
  • [4] D. DeCarlo and J. Gallier. Topological evolution of surfaces. Graphics Interface, pages 194–203, 1996.
  • [5] H. W. Engl, M. Hanke, and A. Neubauer. Regularization of Inverse Problems. Kluwer Academic Publishers, 1 edition, 1996.
  • [6] A. Gregory, A. State, M. C. Lin, D. Manocha, and M. A. Livingston. Interactive surface decomposition for polyhedral morphing. The Visual Computer, 15:453–470, 1999.
  • [7] X. Gu, T.-M. Huang, W.-Q. Huang, S.-S. Lin, W.-W. Lin, and S.-T. Yau. High performance computing for spherical conformal and Riemann mappings. Geometry, Imaging and Computing, 1(2):223–258, 2014.
  • [8] X. Gu, Y. Wang, T. F. Chan, P. M. Thompson, and S.-T. Yau. Genus zero surface conformal mapping and its application to brain surface mapping. IEEE Transactions on Medical Imaging, 8, 2004.
  • [9] X. Gu, Y. Wang, and S.-T. Yau. Geometric compression using Riemann surface structure. Commumications in Information and Systems, 3:171–182, 2004.
  • [10] X. Gu and S.-T. Yau. Computational Conformal Geometry. Higher Education Press, 1 edition, 2008.
  • [11] E. Jeong, M. Yoon, Y. Lee, M. Ahn, S. Lee, and B. Guo. Feature-based surface light field morphing. Proceedings of the 11th Pacific Conference on Computer Graphics and Applications, 2003.
  • [12] S.-Y. Lee, K.-Y. Chwa, J. Hahn, and S. Y. Shin. Image morphing using deformable surfaces. Proceedings of Computer Animation, pages 31–39, 1994.
  • [13] N. Litke, M. Droske, M. Rumpf, and P. Schröder. An image processing approach to surface matching. The Eurographics Association, 2005.
  • [14] Y.-S. Liu, H.-B. Yan, and R. R. Martin. As-rigid-as-possible surface morphing. Journal of Computer Science and Technology, 26:548–557, 2011.
  • [15] D. Martínez, L. Velho, and P. C. Carvalho. Computing geodesics on triangular meshes. Computers and Graphics, 29:667–675, 2005.
  • [16] L. J. van Vliet and P. W. Verbeek. Curvature and bending energy in 2d and 3d images. Scandinavian Conference on Image Analysis, pages 1403–1410, 1993.
  • [17] G. Wolberg. Image morphing: a survey. The Visual Computer, 14:360–372, 1998.
  • [18] M. Yang, K. Wang, and L. Zhang. Realistic real-time facial expressions animation via 3d morphing target. Journal of Software, 8(2), 2013.
  • [19] V. Zanella, G. Ramirez, H. Vargas, and L. V. Rosas. Automatic morphing of face images. 9th International Conference, ICANNGA 2009, 2009.