Geometry and clustering with metrics derived from separable Bregman divergences

1 Introduction and Preliminaries

1.1 Riemannian geometry induced by a Bregman divergence

In [19] it was shown how to put a Riemannian structure on a convex open subset $M$ of $R^{K}$ derived from a Bregman divergence

δ_{Φ} (x, x^{'}) = Φ (x) - Φ (x^{'}) - (x - x^{'})^{⊤} \nabla Φ (x^{'}),

with real-valued generator function $Φ : M \to R$ . There $M = J^{K}$ where $J$ is an interval (bounded or unbounded) in the real line, and the separable Bregman generator $Φ (x) = \sum_{j = 1}^{K} ϕ (x_{j})$ with $ϕ : J \to R$ is a three times continuously differentiable function. This set up was chosen for computational convenience, and $δ_{Φ} (x, x^{'}) = \sum_{j = 1}^{K} δ_{ϕ} (x_{j}, x_{j}^{'})$ . In this case the Riemannian metric on $M$ is given by $g_{i, j} (x) = ϕ^{''} (x_{i}) δ_{i, j}$ .

The Riemannian distance in $M$ induced by that metric is given by

d_{ϕ} (x, x^{'})^{2} = K \sum j = 1 (h (x_{j}) - h (x_{j}^{'}))^{2},

(1.1)

where $h (x) = \int \sqrt{ϕ^{''} (x)}$ is a an increasing continuously differentiable function defined as the primitive of $(ϕ^{''})^{1 / 2}$ (also called antiderivative). Its compositional inverse $h^{- 1}$ will be denoted by $H$ (with $H \circ h^{- 1} = h^{- 1} \circ H = i d$ ) and we shall use the same letter for the extensions of the functions to $M$ , that is, we shall write $h (x)$ to denote the transpose of $(h (x_{1}), . . ., h (x_{K})),$ and the same for $H$ . We term distance $d_{ϕ}$ the Riemann-Bregman distance.

The equation $γ (t) = (γ_{1} (t), \dots, γ_{K} (t))$ of the Riemannian geodesic between $x$ and $x^{'}$ was reported in [19] as $γ_{i} (t) = h^{- 1} (h (x_{i}) + k_{i} t)$ , where the $k_{i} = h (x_{i}^{'}) - h (x_{i})$ are the constants of integration such that $γ (0) = x$ and $γ (1) = x^{'}$ . That is, the geodesics are expressed as

γ_{i} (t) = h^{- 1} ((1 - t) h (x_{i}) + t h (x_{i}^{'})), t \in [0, 1] .

(1.2)

Notice that by introducing the Legendre convex conjugate

	$ϕ^{*} (y)$	$=$	$sup x \in J {y x - ϕ (x)},$		(1.3)
		$=$	$y {ϕ^{'}}^{- 1} (y) - ϕ ({ϕ^{'}}^{- 1} (y)),$		(1.4)

with $y_{i} = ϕ^{'} (x_{i})$ (and $x_{i} = {ϕ^{'}}^{- 1} (y_{i}) = {ϕ^{*}}^{'} (y_{i})$ ), we can write the dual Riemannian metric as ${g^{*}}^{i j} (y) = {ϕ^{*}}^{''} (y_{i}) δ_{i, j}$ , and we have the following identity: $g (x) g^{*} (y) = i d$ .

It follows that we can express the Bregman-Riemannian distance either by using the primal affine $x$ -coordinate system, or by using the dual affine $y$ -coordinate system (with $y = \nabla Φ (x)$ and $x = \nabla Φ^{*} (y)$ ):

Lemma 1.1 (Dual Riemann-Bregman distances)

We have $δ_{Φ} (x, x^{'}) = δ_{Φ^{*}} (y^{'}, y)$ , and

d_{ϕ} (x, x^{'}) = d_{ϕ^{*}} (y, y^{'}) = d_{ϕ^{*}} (\nabla Φ (x), \nabla Φ (x^{'})) .

(1.5)

Example 1.1

Consider $ϕ (x) = x log x - x$ (extended Shannon negentropy). We have $ϕ^{'} (x) = log x$ and $ϕ^{''} (x) = \frac{1}{x}$ . It follows that $h (x) = 2 \sqrt{x} + c_{x}$ , where $c_{x}$ is a constant. The Legendre conjugate is $ϕ^{*} (y) = e^{y}$ and $h^{*} (y) = \int \sqrt{{ϕ^{*}}^{''}} = 2 e^{\frac{y}{2}} + c_{y}$ , where $c_{y}$ is a constant. We have $y (x) = log x$ , $d_{ϕ} (x, x^{'}) = 2 | \sqrt{x} - \sqrt{x^{'}} |$ and $d_{ϕ^{*}} (y, y^{'}) = 2 | e^{\frac{y}{2}} - e^{\frac{y^{'}}{2}} |$ . We check that $d_{ϕ} (x, x^{'}) = d_{ϕ^{*}} (y, y^{'})$ since $e^{\frac{y}{2}} = \sqrt{x}$ .

Figure 1 illustrates the Legendre duality for defining the Riemann-Bregman distances in the dual coordinate systems induced by the Legendre transformation.

Figure 1: The Riemann-Bregman distances expressed in the dual coordinate system induced by the Legendre transformation.

Remark 1.1

In information geometry [2, 13], a smooth dissimilarity measure $D$ on a manifold $M$ induces a dual structure $(M, g_{D}, \nabla_{D}, \nabla_{D}^{*})$ defined by a pair of affine connection $(\nabla_{D}, \nabla_{D}^{*})$

coupled to the metric tensor

g_{D}

. The Riemannian geometry

(M, g_{D})

is a self-dual information-geometric structure with the dual connections coinciding with the Levi-Civita connection

\nabla^{L C}

. Bregman divergences

δ_{Φ}

yield dually-flat information-geometric spaces (meaning

\nabla_{D}

-flat and

\nabla_{D}^{*}

-flat) although the Riemann-Bregman geometry is usual curved [30] (meaning

\nabla_{D}^{L C}

-curved).

1.2 Examples of Riemann-Bregman distances

Let us now list a few examples. These will be used below to exemplify the geometric objects defined by the distances and to exemplify the clustering algorithms.

1.2.1 Case 1: $ϕ = x^{2} / 2$

In this case $ϕ^{''} (x) = 1$ and $ϕ^{'''} (x) = 0.$ The induced distance is the standard squared Euclidean distance

d_{ϕ} (x, y)^{2} = K \sum i = 1 (x_{i} - y_{i})^{2} .

1.2.2 Case 2: $ϕ (x) = e^{x}$

Now $ϕ^{''} (x) = ϕ^{'''} (x) = e^{x} .$ The geodesic distance between $x$ and $y$ is given by

d_{ϕ} (x, y)^{2} = K \sum i = 1 (e^{y_{i} / 2} - e^{x_{i} / 2})^{2} .

1.2.3 Case 3: $ϕ (x) = e^{- x}$

The geodesic distance between $x$ and $y$ is given by

d_{ϕ} (x, y)^{2} = K \sum i = 1 (e^{- y_{i} / 2} - e^{- x_{i} / 2})^{2} .

1.2.4 Case 4: $ϕ (x) = x ln x$ (Shannon negentropy)

This time our domain is $M = (0, \infty)^{K}$ and $ϕ^{'} (x) = (ln x) - 1$ , $ϕ^{''} (x) = \frac{1}{x}$ and $h (x) = 2 \sqrt{(} x)$ . The geodesic distance between $x$ and $y$ is

d_{ϕ} (x, y)^{2} = 2 K \sum i = 1 (\sqrt{y_{i}} - \sqrt{x_{i}})^{2} .

This looks similar to the squared Hellinger distance used in probability theory. See Pollard’s

[31]

1.2.5 Case 5: $ϕ (x) = - ln x$ (Burg negentropy)

To finish, we shall consider another example on $M = (0, \infty)^{K}$ . We have $ϕ^{'} (x) = - \frac{1}{x}$ , $ϕ^{''} (x) = \frac{1}{x^{2}}$ and $h (x) = log x + c$ . Now, the distance between $x$ and $y$ is now given by

d_{ϕ} (x, y)^{2} = K \sum i = 1 (ln y_{i} - ln x_{i})^{2} .

To continue, let us mention that the thrust in [19]

was to examine the prediction process when considering random variables taking values in metric space

(M, d_{ϕ}),

that is to examine the concepts of best predictor of a random variable

X

taking values in

M

when the prediction error is measured in the

d_{ϕ} -

distance. The special relation of the distance (1.1) to the Euclidean distance makes the following result clear:

Theorem 1.1 (Separable Riemann-Bregman centroid)

Given a collection of points ${x_{1}, . . ., x_{N}}$ in $M,$ the point $^x$ the realizes the minimum of

N \sum n = 1 d_{ϕ} (x_{n}, ξ)^{2} % o v e r ξ \in M

is unique and given by

^x = H (\frac{1}{N} N \sum n = 1 h (x_{n})) .

(1.6)

Observe that formula Eq. 1.6 coincides with the left-sided Bregman centroid [28]. However, left-sided Bregman $k$ -means and Riemann-Bregman $k$ -means will differ in the assignment steps because they rely on different dissimilarity measures.

We shall make extensive use of this definition when computing the centers of the clusters in the various metrics. This is done in Section 4 below, where we start from a given data set and organize it in clusters according to different clustering procedures for each of the distances, and compute the centers of the resulting clusters each distance as in (1.6).

In Section 2 we consider some standard geometric objects, like Balls, Voronoi cells, in the different distances, whereas in Section 3 we consider the problem of fixed rate quantization for $M -$ valued random variables .

2 Balls and Voronoi cells in the derived metrics

2.1 Balls

The examples listed show that $h : (M, d_{ϕ}) \to (R^{K}, ∥ \cdot ∥)$ is a non-linear isometry. This allows us to describe balls in $(M, d_{ϕ})$ as

B_{ϕ} (x, r) = {y \in M : d_{ϕ} (y, x) \leq r} = h^{- 1} (B (h (x), r)) .

(2.1)

Here, $B (h (x), r)$ denotes the ball in $R^{K},$ of radius $r$ centered at $h (x)$ . The space of balls can also be defined using a potential function following [10].

2.2 Voronoi diagrams

Let $S = {y_{1}, . . ., y_{n}}$ be some subset $(M, d_{ϕ}) .$ The Voronoi cell in $M$ of $y_{i}$ is defined by

{v o r}_{ϕ} (y_{i}) = {x \in M : d_{ϕ} (x, y_{i}) \leq d_{ϕ} (x, y_{j}), \forall y_{j} \in M} .

(2.2)

And as above, if we denote by $v o r (h (y_{i}))$ the Voronoi cell of $h (y_{i})$ in the Euclidean metric, it is clear that

{v o r}_{ϕ} (y_{i}) = h^{- 1} (v o r (h (y_{i}))) .

(2.3)

A few graphical examples are shown in Figure 2.

Figure 3 displays four types of Voronoi diagrams induced by a Bregman generator; The pictures illustrate the fact that the symmetrized Bregman Voronoi diagram is different from the (symmetric metric) Riemann-Bregman Voronoi diagram.

3 Fixed rate quantization

Let $M \subseteq R^{K}$ be provided with a metric $d_{ϕ}$ derived from a divergence. Let $N \geq 1$ be a given integer and $C = {y_{1}, . . ., y_{N}} \subset M$ be a given set, called the codebook and its elements are called the

code vectors

. A fixed rate quantizer (with codebook

C

) is a measurable mapping

q : M \to C .

The cardinality of

C

is called the quantization rate. Two quantizers have the same rate if their codebooks have the same cardinality

For a $M -$ valued random variable $X$ defined on a probability space $(Ω, F, P),$ the $ϕ -$ distortion of $q$ in quantizing $X$ is given by the expected reconstruction square error:

D_{ϕ} (X, q (X))^{2} = E [d_{ϕ} (X, q (X))^{2}] .

(3.1)

If we are interested in only one $M -$ valued random variable $X,$ we may replace $Ω$ by $M,$ think of $F$ as the Borel sets of $M,$ think of $X$ as the identity mapping, and think of $P$ as a probability measure on $(M, F) .$ Also, our notation is slightly different from that of Linder [24] in that we put a square into the definition of the square error. Since $q (X)$ is constant on the “level” sets $S_{i} := {q (X) = y_{i}}$ for $i = 1, . . ., N,$ the former expectation can be written as

D_{ϕ} (X, q (X))^{2} = E [E [d_{ϕ} (X, q (X))^{2} | σ (q)]] .

Here we denote by $σ (q)$ the sub $- σ -$ algebra of $F$ generated by the partition ${S_{1}, . . ., S_{N}}$ of $Ω$ induced by the quantizer $q .$ When $P (S_{i}) > 0$ holds for all blocks of the partition, the conditional expectation takes values

E [d_{ϕ} (X, q (X))^{2} | σ (q)] = \frac{1}{P (S_{i})} \int_{S_{i}} d_{ϕ} (X, y_{i})^{2} d P on the set S_{i}, i = 1, . . ., N .

(3.2)

Notice that given $σ (q),$ there is a quantizer having the same rate and realizing an error smaller than $D_{ϕ} (X, q (X))^{2} .$ The existence and uniqueness of such quantizer is given by the following result in [19].

Theorem 3.1

Let $G$ be a sub $- σ -$ algebra of $F .$ Let $X$ be such that $h (X) \in L_{2}$ (Lebesgue space of functions that are square integrable). Then there exists a unique $G$ measurable random variable, denoted by $E_{ϕ} [X | G]$ satisfying

E_{ϕ} [X | G] = argmin {D_{ϕ} (X, Y)^{2} | Y \in G, h (Y) \in L_{2}}

The result mentioned a few lines above can be stated as

Theorem 3.2

Let $q$ be a quantizer of rate $N .$ Suppose that the blocks ${S_{i} : i = 1, . . ., N}$ of the partition of $Ω$ determined by $q$ are such that $P (S_{i}) > 0$ for all $i = 1, . . ., N .$ Then

q^{*} (X) = E_{ϕ} [X | G] = N \sum i = 1 \frac{1}{P (S_{i})} E_{ϕ} [X; S_{i}] I_{S_{i}} (X) := N \sum i = 1 y_{i} I_{S_{i}} (X) .

(3.3)

is the quantizer with codebook $C^{*},$ whose elements are $y_{i}^{*} = \frac{1}{P (S_{i})} E_{ϕ} [X; S_{i}],$ which makes the error $D_{ϕ} (X, q (X))^{2}$ smaller over all codebooks which determine the same partition of $Ω .$

An interesting way to associate sets to a codebook $C = {y_{1}, . . ., y_{N}}$ of rate $N$ is by means of Voronoi cells. ${S_{1}, . . ., S_{N}}$ are the Voronoi cells determined by $C,$ their intersections are polyhedra of dimension lower than $K$ (faces), and the probability $P$ does not change them, that is if $P (S_{i} \cap S_{j}) = 0$ whenever $i \neq j .$ If we put $B_{i} = int (S_{i})$ for the interior of the Voronoi cells, they generate a sub- $σ -$ algebra of $F$ such that $\cup_{i} B_{i}$ differs from $M$ by a set of $P -$ probability $0.$

In the notation of Theorem 3.2 we have

Corollary 3.1

If the $σ -$ algebra $G$ is generated buy the interiors of the Voronoi cells determined by a codebook $C,$ then

y_{i}^{*} = \frac{1}{P (S_{i})} E_{ϕ} [X; S_{i}] = \frac{1}{P (B_{i})} E_{ϕ} [X; B_{i}] \in B_{i} \subset S_{i}, \forall i = 1, . . ., N .

The proof is clear: The integral is a weighted average of points in a convex set.

4 Riemann-Bregman Clustering

In this section we shall consider a $2 -$ dimensional data set in which several clusters (indicated) by different colors in the figures displayed below are apparent. We shall suppose that the base space in which the random variable takes values comes equipped with one of the five distances listed in Section 1.2.

We shall suppose that the points belong one of the possible manifold on which a Riemann-Bregman metric has been defined, and the object of the exercise is to identify the clusters. To assign the data points into clusters, we shall apply the $k$

-means, the EM (Expectation-Maximization), the HCPC (Hierarchical Clustering Principal Component) and the HAC (Hierarchical Agglomerative Clustering) techniques. Once the clusters are identified, we determine their centers by invoking Theorem (

1.1 and (1.6). In the four examples described below, we shall see how the choice of the metric determines the clusters.

The points were generated all in $(0, \infty)^{2}$ in order to be able to visually compare the result of the clustering procedures in the distances listed on Section 1.2.

Strategy:
Step 1 Generate a cloud ${x_{1}, . . . ., x_{n}}$ in $(0, \infty)^{2},$
Step 2 Apply the $h -$ mapping and form the cloud ${h (x_{1}), . . ., h (x_{N})}$ in $h (M) .$
Step 3 Apply the clustering algorithms. If ${η_{1}, . . ., η_{M}}$ are the clusters of the new cloud, then the clusters in the original coordinates are ${y_{1} = h^{- 1} (η_{1}), . . ., y_{n} = h^{- 1} (η_{n})}$ .
Step 4 Plot the “centers” ${y_{1}, . . ., y_{n}}$ in the original cloud.

Notice that in all examples, the mappings $h$ and $h^{- 1}$ are invertible, and are applied componentwise. The aim of the exercise is to see how the choice of the metric determines the clusters.
In particular, we observe that although the centroid update rule coincide with the left-sided Bregman clustering [28], the assignment steps of the left-sided Bregman clustering and the Riemann-Bregman clustering are different.

4.1 The data set

For all the the numerical experiments considered below, the “original” data is composed of samples of four pairs of independent normal variables with parameters specified in Table 1 shown next.

Cluster		$X_{1}$	$X_{2}$
1	$σ_{1}$	0.5	0.5
1	$μ_{1}$	8	6.5
2	$σ_{2}$	0.4	0.45
2	$μ_{2}$	9	7.5
3	$σ_{3}$	0.35	0.35
3	$μ_{3}$	8.5	9
4	$σ_{4}$	0.6	0.6
4	$μ_{4}$	8	10

Table 1: Centers of

k

-means clusters for a groundtruth synthetic dataset.

Thus we obtain four point clouds with some overlap. The number of points in the four original groups is, respectively, $100, 300, 200, 150.$ These numbers will be used as reference to check upon the performance of each clustering technique when the distance between points is measured in each of the five different Riemann-Bregman metrics.

4.2 Clustering by $k$ -means

The k-means clustering algorithm is an iterative methodology that assigns data points to a cluster, in such a way that the sum of the squared distances between the data points and the cluster’s centroid (defined as the arithmetic mean) is minimal. The less variation we have within clusters, the more homogeneous the data points are within the same cluster.

In Figure 4 we present the result of the application of the $k -$ means procedure to the original data set depicted in panel (a)a. The caption of each of the panels describes the $h -$ function that has been applied to the data before applying the clustering procedure. Once the clusters are determined, the in-cluster mean is computed and the whole set is mapped back to the original space.

We add at this point that if one suspects (or knows) that the data sets are in different scales, it might be convenient to standardize the data. This is important since clustering algorithms are based on distance comparison procedures. In [5], Banerjee et al. highlight the bijection between regular exponential families and a class of Bregman divergences called regular Bregman divergences. They demonstrate experimentally that the $k$ -means clustering perform best in practice on mixture of exponential families (with cumulant function $ϕ$ ) when the Bregman divergence is chosen for the dual convex conjugate $ϕ^{*}$ .

The centers of the original data set and those of the clusters are listed in Table 2.

Cluster	1	2	3	4
original	(7.93, 6.63)	(9.01, 7.51)	(8.5, 9.08)	(8.01, 10.01)
$h (x) = x$	(7.97, 6.6)	(9.04, 7.53)	(8.5, 9.08)	(7.93, 10.15)
$h (x) = \sqrt{x}$	(7.97, 6.58)	(9.02, 7.53)	(8.52, 9.09)	(7.9, 10.12)
$h (x) = ln (x)$	(7.96, 6.56)	(9.02, 7.52)	(8.55, 9.1)	(7.87, 10.07)
$h (x) = exp (0.5 x)$	(8.27, 7.06)	(9.24, 7.6)	(8.38, 9.18)	(8.05, 10.32)
$h (x) = exp (- 0.5 x)$	(7.76, 6.19)	(8.6, 7)	(9.05, 7.74)	(8.29, 9.49)

Table 2: centers of

k

-means clusters obtained with different methodologies

From the table it is clear that the Euclidean distance and the logarithmic distance perform better than the others in capturing the original centers of the data. Actually, the centers of the original data are the minimizers to the original data set in the Euclidean distance and are the empirical means of the normals used to generate the data.

As a further test of the clustering algorithm we provide a “headcount” of points in each cluster and compare the numbers with the number of original points in each cluster. The result is displayed in Table 3.

Cluster	1	2	3	4	Accuracy
Original	100	300	200	150	1
$h (x) = x$	109	286	226	129	0.953
$h (x) = \sqrt{x}$	106	289	225	130	0.959
$h (x) = ln (x)$	103	292	221	134	0.968
$h (x) = exp (0.5 x)$	214	196	240	100	0.768
$h (x) = exp (- 0.5 x)$	46	146	211	347	0.388

Table 3: Size of the

k

-means clusters and accuracy obtained with different transformations

4.3 Clustering with the EM algorithm

The expectation maximization (EM) algorithm [17]

is an iterative method to estimate the parameters in a statistical model, in which the model depends on unobserved variables. The EM iteration alternates between performing an expectation (E) step and a maximization (M) step. In the first step a likelihood function is created and in the second it is maximized. See

[26] for details.

In the EM methodology an initial guess is made for the model’s parameters and a distribution is created, this is the E-step or Expected distribution, then the parameters are updated in the (M-step) maximizing the expectation computed in the E-step. Both steps are repeated until the convergence is clear (distribution that doesn’t change from the E-step to the M-step)

In the upper left panel of Figure 5 we display the original clusters (the same as above) for the sake of easy visual comparison, as well as the clusters obtained by applying the EM clustering methodology.

Form Table 5 it becomes clear how the distance between points changes plus the type of clustering algorithm determine the centers of the clusters.

Cluster	1	2	3	4
Original	(7.93, 6.63)	(9.01, 7.51)	(8.5, 9.08)	(8.01, 10.01)
$h (x) = x$	(7.82, 6.52)	(8.99, 7.49)	(8.47, 9.2)	(7.79, 10.29)
$h (x) = \sqrt{(} x)$	(7.72, 6.41)	(8.94, 7.44)	(8.26, 9.32)	(8.49, 10.53)
$h (x) = ln (x)$	(7.95, 6.55)	(9.01, 7.53)	(8.55, 9.07)	(7.95, 10.03)
$h (x) = exp (0.5 x)$	(7.79, 6.59)	(8.88, 7.37)	(9.55, 7.67)	(8.31, 9.34)
$h (x) = exp (- 0.5 x)$	(7.92, 6.6)	(9.01, 7.53)	(8.52, 9.1)	(7.93, 10.09)

Table 4: centers of EM clusters in the different metrics

In Table (5

), we list the size of the original clusters along with the size of the clusters obtained in the different distances. Also, in the last column of the table we list the accuracy of the procedure, defined as the number of cases correctly classified. Obviously, with the original data all the cases are correct, but note that since the distance between the points changes with the metric, it is natural that the accuracy of the procedure would vary.

Cluster	1	2	3	4	Accuracy
Original	100	300	200	150	1
$h (x) = x$	85	315	259	91	0.901
$h (x) = \sqrt{(} x)$	64	333	307	46	0.809
$h (x) = ln (x)$	101	298	200	151	0.996
$h (x) = exp (0.5 x)$	83	217	69	381	0.577
$h (x) = exp (- 0.5 x)$	101	303	208	138	0.977

Table 5: Size of the EM clusters and accuracy for the different metrics

5 Clustering using the HCPC algorithm

The hierarchical clustering on principal components (HCPC) algorithm is a combination of methods. First it uses principal component analysis (PCA) to reduce possible hidden high dimensionality in the data. Then it uses hierarchical and partitional clustering to describe similarity between the points. See

[20] for details and further references.

In Figure 6 we display the results of the clustering procedure. The panels are as displayed as above. In the upper leftmost one, appears the original data, and the remaining panels we display the results of the clustering process. The resulting clusters are in different colors.

In Table 6 we list the centers of the HCPC-clusters in the different distances.

Cluster	1	2	3	4
original	(7.93, 6.63)	(9.01, 7.51)	(8.5, 9.08)	(8.01, 10.01)
$h (x) = x$	(7.9, 6.71)	(9.06, 7.49)	(8.54, 9.26)	(7.67, 9.93)
$h (x) = \sqrt{(} x)$	(7.89, 6.71)	(9.06, 7.48)	(8.55, 9.26)	(7.67, 9.91)
$h (x) = ln (x)$	(7.87, 6.67)	(9.05, 7.47)	(8.55, 9.25)	(7.66, 9.9)
$h (x) = exp (0.5 x)$	(7.96, 7.52)	(9.38, 7.71)	(8.77, 8.04)	(8.04, 9.98)
$h (x) = exp (- 0.5 x)$	(7.85, 6.5)	(9.02, 7.46)	(8.51, 9.29)	(7.54, 9.79)

Table 6: centers of the HCPC clusters in the different metrics

In Table 7, we can observe the size of the original clusters along with the size obtained with the transformations. Also, this table contains a column that is the accuracy, this value is defined as the number of cases correctly classified. Obviously, with the original data all the cases are correct, for that reason we have a accuracy of 1, but for the other transformations we fail.

Cluster	1	2	3	4	Accuracy
original	100	300	200	150	1
$h (x) = x$	106	290	247	107	0.929
$h (x) = \sqrt{(} x)$	105	288	250	107	0.927
$h (x) = ln (x)$	100	289	256	105	0.925
$h (x) = exp (0.5 x)$	166	135	276	173	0.749
$h (x) = exp (- 0.5 x)$	83	297	285	85	0.864

Table 7: Size of the HCPC clusters and accuracy obtained with different transformations

6 The Hierarchical Agglomerative Clustering (HAC) process

The Hierarchical Agglomerative Clustering (HAC) is designed to build a hierarchy of clusters using a bottom-up or agglomerative strategy. Here each data point is treated as a single cluster and then a distance (or similarity) between each of the clusters is computed in order to merge two or more into one (e.g., single linkage or average linkage), that is, the two most “similar” clusters are merged at each step. See [21] and [30] for details.

In Figure 7 we display the results of the hierarchical clustering algorithm. The panels are organized as above.

In Table 8 we list the centers associated to the resulting clusters. As usual, they are the $d_{ϕ} -$ arithmetic means of the clusters provided by the hierarchical clustering algorithm.

Cluster	1	2	3	4
original	(7.93, 6.63)	(9.01, 7.51)	(8.5, 9.08)	(8.01, 10.01)
$h (x) = x$	(8.16, 6.71)	(9.06, 7.61)	(8.28, 9.26)	(8.34, 10.15)
$h (x) = \sqrt{(} x)$	(8.38, 6.92)	(9.22, 7.68)	(8.41, 9.31)	(7.38, 10.29)
$h (x) = ln (x)$	(7.86, 6.69)	(9.04, 7.44)	(8.56, 9.09)	(7.86, 9.98)
$h (x) = exp (0.5 x)$	(8.72, 7.31)	(8.32, 9.42)	(8.07, 10.84)	(8, 11.59)
$h (x) = exp (- 0.5 x)$	(7.99, 5.89)	(8.67, 7.1)	(8.57, 9.01)	(7.23, 10.14)

Table 8: centers of the clusters provided by the hierarchical clustering

In Table 9, we list the size of the original clusters along with the sizes of the clusters in the different metrics. We also list the accuracy of the classifier.

Comment At this point we mention that in all examples the accuracy can be provided because we new the true clusters before hand. Therefore, this accuracy measure is an indirect quality test of the different clustering methods. Other measures of cluster quality can be considered (e.g., the adjusted Rand index or the Normalized Mutual Information [5]).

Cluster	1	2	3	4	Accuracy
original	100	300	200	150	1
$h (x) = x$	144	256	261	89	0.860
$h (x) = \sqrt{(} x)$	215	170	323	42	0.683
$h (x) = ln (x)$	94	291	226	139	0.957
$h (x) = exp (0.5 x)$	407	325	13	5	0.140
$h (x) = exp (- 0.5 x)$	19	263	435	33	0.579

Table 9: Size of the Hierarchical clustering and accuracy obtained with different transformations

7 Concluding comments

In this note, we considered the Riemannian metric distance induced by a separable Bregman divergence, and studied the fixed rate quantization problem. We concisely described the Riemann-Bregman Voronoi diagrams that can be obtained from an ordinary Euclidean Voronoi diagram after a monotone isometric embedding. Then we reported on the partition-based ( $k$ -means), soft (EM) and hierarchical (HCPC/HCA) clusterings with respect to these Riemann-Bregman distances. As was to be expected, the resulting clusters depend on the clustering method as well as on the metric used to measure the distance between points.

References

[1] Amari, S and Cichocki, A. (2010). Information theory of divergence functions, Bull. of the Polish Acad. Sci., 58, 183-195.
[2] Amari, S. and Nagaoka, H. (2000). Methods of Information Geometry, Transl. of Mathem. Monographs, 191, Oxford Univ. Press, New York.
[3] Banerjee, A., Guo, X. and Wang, H. (2005). On the optimality of conditional expectation as Bregman predictor, IEEE Transactions on Information Theory, 51, 2664 - 2669
[4] Banerjee, A. Dhillon, I., Ghosh, I. Merugu, S and Modhi, D.s. (2008). A Generalized Maximum Entropy Approach to Bregman Co-clustering and Matrix Approximation
The Journal of Machine Learning Research,
8, 1919-1986.
[5] Banerjee, A., Merugu, S., Dhillon, I. S., and Ghosh, J. (2005) Clustering with Bregman divergences. Journal of machine learning research 6: 1705-1749.
[6] Baushke, H.H. and Borweinn, J.M. (1997). Legendre Functions and the Method of Random Bregman Projections, Journal of Convex Analysis, 4, 27-67.
[7] Baushke, H.H and Lewis, A. (1998). Dykstras algorithm with Bregman projections: A convergence proof, Optimization, 48, 409-427
[8] Baushke, H.H., Borwein, J.M and Combettes, P.L. (2003). Bregman monotone optimization algorithms, SIAM J. Control Optim., 42, 596-636.
[9] Baushke, H.H and Combettes, P.L. (2003). Construction of best Bregman approximation in reflective Banach spaces, Proc. Amer. Math. Soc., 131, 3757-3766
[10] Boissonnat, J-D., Nielsen, F and Nock, F. (2010). Bregman Voronoi Diagrams, Discrete Comput. Geom., 44, 281-307.
[11] Bregman, L. (1967). The relaxation method of finding common points of convex sets and its application to the solution of problems in convex programming, Comp. Math. Phys., USSR, 7, 2100-217.
[12] Butnariu, D. and Resmerita, E. (2005). Bregman distances, totally convex functions, and a method for solving operator equations, Abstract and Applied Analysis, 2006, Article ID 84919, 1-39.
[13] Calin, O. and Urdiste, C. Geometric Modeling in Probability and Statistics, Springer Internl. Pub., Switzerland, (2010).
[14] Censor, Y and Reich, S. (1998). The Dykstra algorithm with Bregman projections. Comm. Applied Analysis, 2, 407-419.
[15] Censor, Y. and Zaknoon, M. (2018). Algorithms and convergence results of projection methods for inconsistent feasibility problems: A review, arXiv:1802.07529v3 [math.OC].
[16] Csiszár, I. (2008). Axiomatic characterization of information measures, Entropy, 10, 261-273.
[17] Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society. Series B (methodological), 1-38.
[18] Fisher, A. (2010). Quantization and clustering with Bregman divergences
, Journal of Multivariate Analysis,
101, 2207-2221.
[19] Gzyl, H. (2018). Prediction in Riemannian metrics derived from divergence functions, http://arxiv.org/abs/1808.01638.
[20] Husson, F, Josse, J. and Pagès J. (2010). ——Principal Component Methods - Hierarchical Clustering - Partitional Clustering: Why Would We Need to Choose for Visualizing Data?” Technical Rept, Mathem. and Stats Dept., Univ Rennes.
[21]
Kassambara, A. (2017). Practical Guide To Cluster Analysis in R. CreateSpace: North Charleston, SC, USA.
[22] Lang, S. Math talks for undergraduates, Springer, New York, (1999).
[23] Li, C., Song, W. and Yao, J-C. (2010). The Bregman distance, approximate compactness and convexity of Chebyshev sets in Banach spaces,Journal of Approximation Theory 162, 1128-1149.
[24] Linder, T. (2001). Learning theoretic methods in vector quantization, in Principles of Nonparametric Learning, I. Gyorfi, ED., CISM Lecture Notes, Springer, New York.
[25] Lawson, J.D. and Lim, Y. (2001)
The Geometric mean, matrices, metrics and more
, Amer. Math.,Monthly, 108, 797-812.
[26] McLachlan, G.J. and Peel, D. (2000). Finite Mixture Models, Wiley, New York.
[27] Moahker, M. (2005) A differential geometric approach to the geometric mean of symmetric positive definite matrices, SIAM. J. Matrix Anal. & Appl., 26, 735-747.
[28] Nielsen, F., and Nock, R. (2009). Sided and symmetrized Bregman centroids. IEEE transactions on Information Theory 55.6: 2882-2904.
[29] Nielsen, F. (2016). Hierarchical Clustering in
Introduction to HPC with MPI for Data Science
Springer, Berlin.
[30] Nielsen, F. (2018). An elementary introduction to information geometry. Available at https://arxiv.org/abs/1808.08271
[31] Pollard, D. (2002). A user’s guide to measure theoretic probability, Cambridge Univ. Press., Cambridge.
[32] Schwartzmazn, A. (2015) Lognormal distribution and geometric averages of positive definite matrices, Int. Stat. Rev., 84, 456-486.

Geometry and clustering with metrics derived from separable Bregman divergences

Authors

Hodge Theory on Metric Spaces

On the Metric Distortion of Embedding Persistence Diagrams into separable Hilbert spaces

Every metric space is separable in function realizability

Embeddings of Persistence Diagrams into Hilbert Spaces

Prediction in Riemannian metrics derived from divergence functions

Geometric clustering in normed planes

A survey of mass partitions

1 Introduction and Preliminaries

1.1 Riemannian geometry induced by a Bregman divergence

Lemma 1.1 (Dual Riemann-Bregman distances)

Example 1.1

Remark 1.1