Data Interpolations in Deep Generative Models under Non-Simply-Connected Manifold Topology

01/20/2019 ∙ by Jiseob Kim, et al. ∙ Seoul National University 0

Exploiting the deep generative model's remarkable ability of learning the data-manifold structure, some recent researches proposed a geometric data interpolation method based on the geodesic curves on the learned data-manifold. However, this interpolation method often gives poor results due to a topological difference between the model and the dataset. The model defines a family of simply-connected manifolds, whereas the dataset generally contains disconnected regions or holes that make them non-simply-connected. To compensate this difference, we propose a novel density regularizer that make the interpolation path circumvent the holes denoted by low probability density. We confirm that our method gives consistently better interpolation results from the experiments with real-world image datasets.



There are no comments yet.


page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


Deep generative models such as Generative Adversarial Network (GAN) goodfellow2014generative and Variational Auto-Encoder (VAE) kingma2013auto show remarkable performance in learning the manifold structure of the data. Among other advantages, knowledge of the manifold structure allows interpolating the data points along the manifold, which gives semantically plausible interpolation results.

Toward geometrically grounded interpolations on the manifold, some recent works shao2017riemannian,chen2018metrics proposed a geodesic interpolation method. Unlike previous methods which naïvely followed linear paths in the latent space, geodesic interpolation methods find the shortest paths on the manifold based on the metric induced from the ambient data-space. This shares the philosophy with classic algorithms like Isomap tenenbaum2000global, and well-grounded on the Riemannian geometry. However, when experimented with real-world data, it often gives poor results containing unrealistic interpolation points.

We argue that the problem stems from a topological mismatch between the model and the data. Deep generative models define a simply-connected manifold because the latent space is endowed with a probability distribution of simply-connected support, and the smooth generative mapping

111The smoothness of the generative mapping is a necessary condition to use the geodesic interpolation methods. preserves the latent-space topology in the data-space. On the other hand, real-world datasets generally contains disconnected regions or holes which make them non-simply-connected. As this topological difference is a fundamental matter, deep generative models cannot help including the holes in the dataset as valid regions in their manifold representation. In consequence, the geodesic curve often finds a shortcut that passes through the holes and the interpolation points get unrealistic.

To tackle this problem, we propose to add a density regularization term to the path-energy loss. The idea comes from an observation that even though deep generative models cannot be matched to the dataset topology, the maximum-likelihood training forces the models to have low probability densities where the holes exist. Therefore, if a low-density-penalizing term is added to the path-energy loss, minimizing the loss will find a path close to the geodesic while going around the holes denoted by low densities. We demonstrate this method is effective and gives semantically better interpolation results from the experiments.

Geometric Interpolations with Density Regularizer

Figure 1: The true underlying structure of the data is generally non-simply-connected (blue), whereas the manifold induced by the deep generative model is simply-connected (red). This topological difference degrades the quality of the interpolation computed on the deep generative model.


We denote the latent space as , the input data space as and the coordinates of each space as and

respectively. We do not use bold letter for the vectors or matrices, but brackets

are used when explicit indication of matrix is needed.

Figure 2: (a) 2-D circular dataset (b) GAN (2-20-20-2) trained on the dataset. Transformed 2-D latent space grid is shown. Note that the grid also covers the hole in the center due to the topological constraint. (c) shown in coordinates with heat map (i.e., ). The central area of the latent space is mapped to the hole in the input space, thus has a low density. (d) Comparison of the interpolation methods333Note that the trained generative mapping is not injective and not an embedding at some regions. We selected two points that the pathes connecting them do not cause any problems regarding this.Blue-solid: linear interpolation method found in previous literature; Red-dotted: geodesic interpolation method; Green-dash-dotted: geodesic with density regularizer.

Gradient Descent Method to Compute the Geodesic Interpolation

In this section, we introduce a path-energy loss function and corresponding gradient descent method to compute the geodesic curve between two data points.

As the manifold of deep generative model is defined via the latent space and the generative mapping, curves on the manifold can be parameterized in the latent space. The length is measured using the Riemannian metric , which is induced from the Euclidean metric of the ambient data space . The mathematical relationship can be written as using Einstein notation and the Kronecker delta, where the details can be found in chen2018metrics,shao2017riemannian.

Now, let be a curve in a latent space with boundary conditions and . The length of under metric is given as . Instead of directly minimizing the length , minimizing the energy gives the same result yet in an algebraically simpler form, by noting that if 444Can be easily derived using Cauchy-Schwarz Inequality.. Thus the formulation to obtain the geodesic is reduced to

To minimize the energy , we compute the gradient according to a small change to the curve ( with ). This is given as a Gâteaux derivative: , which is the functional analogue of directional derivative.

Note that the Euler-Lagrange equation is derived in a process of finding regardless of , but it is intractable to solve since the metric

described by neural network is too complicated. We instead seek the steepest gradient change

, which is derived using the Cauchy-Schwarz inequality,

where is the Christoffel Symbol. Note that is purely related to the local change of the coordinate bases, thus it becomes zero in the input-space coordinates. Also, the metric is described as Kronecker delta in input-space coordinates, so the equation is far simplified as

where is a small change to the curve described in the input-space coordinates.

To summarize, for a curve in , we transform every point on the curve to the input space using the generative mapping, from which the corresponding curve in is obtained. Then, we compute the acceleration to get the steepest gradient descent change toward minimizing the path-length of the curve. By iteratively updating ( is a learning rate), will converge to the shortest path. Finally, putting back to the latent space will give the geodesic in the latent space. More detailed algorithm can be found in shao2017riemannian and the extended version with our proposing regularizer will be described in the next section.

Density Regularizer

As introduced earlier, the topological difference between the deep generative model and the dataset brings a serious problem in the interpolation. For example, consider a circular dataset and the geodesic between two points in the dataset (Fig. 3). It is natural to think that the geodesic should follow a circular path between two points and pass through dense regions. However, as the hole in the center is a valid region for deep generative models, the geodesic becomes a linear path and passes through the hole (Red-dotted curve in Fig. 3 (d)).

In order to prevent the geodesic from finding this kind of shortcuts, we propose to add a density regularizer term to the energy functional:

where is a regularization weight. The regularization term is in fact an integrated negative log-probability-density along the curve, thus it keeps the curve from passing through low-density regions.

In Fig. 3 (c) it can be seen that the density is low in the central area of the latent-space, which is mapped to the hole of the input data space. Introduction of the regularizer results in an interpolation avoiding this low-density regions, while keeping its path-length as short as possible (Green-dash-dotted curve in Fig. 3 (d)).

Numerical Algorithm for Geodesic Interpolation

As explained in the previous section, geodesic curve can be computed by iteratively updating every point on the curve toward the steepest variational direction . In practice, we approximate the curve with a set of points . As for , we first compute using finite difference method, then pull back the vectors to the latent space using the pseudo-inverse of the Jacobian matrix. The contribution from the gradient of the density regularizer is added to the obtained .

Initial ordered sequence
Compute , for
while not converged do
     Compute , for
end while
Algorithm 1 Geometric Interpolation with Density Regularizer


Datasets, Models and Hyperparameters

In experiments, we use two different image datasets: 1) MNIST lecun1998gradient, and 2) Yale belhumeur1997eigenfaces. Yale dataset originally consists of face images of 28 individuals, but only a single person images (under 64 different illumination conditions) are used to examine the illumination manifold.

We use three different deep generative models, where each of which is paired with the datasets. 1) MNIST with GAN goodfellow2014generative, 2) MNIST with VAE kingma2013auto, 3) Yale with DCGAN (Deep Convolutional GAN) radford2015unsupervised.

When running the numerical algorithm to compute the geodesic, number of approximation points is fixed to 35. We observed that too large number of approximation points sometimes yield an entire break-up of the smoothness of the curve due to the approximation noise. Regularizer weight is empirically determined, ranging from 0.1 ~ 0.003 according to each setting.

Figure 3: Interpolation results from MNIST GAN, MNIST VAE and Yale DCGAN models, using different methods. In all plots, colors are matched according to the interpolation methods: Geodesic with Density Regularizer (green), Geodesic (red), Straight in (blue). Aside from the colors, the images are organized in order of GeodReg, Geod, StraightZ, from top to bottom. Corresponding distances are shown in the left with striped, dotted, plain bar graphs. Negative Log-Likelihood along the interpolation points on the path are shown at the bottom with dash-dotted, dotted, plain curves.

Interpolations and the Effect of the Regularizer

In Fig. 3, we show the results from geodesic interpolation method, with and without the density regularizer, as well as the results from a naïve method that follow a straight line in . Hereafter, we denote each method as GeodReg, Geod and StraightZ respectively.

In MNIST results, it can be clearly seen that GeodReg outperforms others in terms of quality. GeodReg shows smooth transitions along realistic samples, whereas Geod and StraightZ contain some mottled, non-number-like images. Geod and StraightZ do not consider the probability density, thus cannot avoid low density regions as shown in the log-density plot in the bottom. Geod particularly passes through much lower density regions, but instead has the shortest length among others. Note that these results agree with the toy example shown in Fig. 3 (d).

Lambertian Image Manifold

According to basri2003lambertian, object images taken under various illumination conditions form a 9-D linear manifold if the object has Lambertian reflectance. Yale face dataset matches up to the Lambertian condition in practice lee2005acquiring, thus we examine if our interpolation methods find linear paths on Yale dataset.

In the left pane of the Fig. 4, it can be seen that the length obtained from Geod is shortest thus closest to the true length. In the right pane, it can be seen that the path found by Geod is the straightest in the input space, as it has the smallest cosine dissimilarity to the true linear path. GeodReg deviates from the true path while seeking high density but still close to it compared to StraightZ as seen from the right pane.

Figure 4: Results from Yale dataset. (Left) Lengths of the curves obtained from different interpolation methods, compared to the ground truth length. (Right) Cosine dissimilarities between the tangent vectors at each point on the curve and the direction of the true linear path are shown. In both subfigures, Geod finds the shortest and the straightest one, thus closest to the ground truth. Interpolations for 13 different sample pairs are shown.

Concluding Remarks

In this paper, we have proposed a density-regularized geometric interpolation method for deep generative models trained from non-simply-connected datasets. The density regularizer has played a crucial role to compensate the topological difference between the model and the data. Our method has shown superior interpolation qualities over previous linear and geodesic interpolation methods. This work was partly supported by the Korea government (2015-0-00310, 2017-0-01772, 2018-0-00622, KEIT-10060086).