K-medoids Clustering of Data Sequences with Composite Distributions

I Introduction

THIS paper aims to cluster sequences generated by unknown

continuous distributions into classes so that each class contains all the sequences generated from the same (composite) distribution cluster. By sequence, we mean a set of feature observations generated by an underlying probability distribution. Here each distribution cluster contains a set of distributions that are close to each other, whereas different clusters are assumed to be far away from each other based on a certain distance metric between distributions. To be more concrete, we assume that the maximum intra-cluster distance (or its upper bound) is smaller than the minimum inter-cluster distance (or its lower bound). This assumption is necessary for the clustering problem to be meaningful. The distance metrics used to characterize the difference of data sequences and corresponding distributions are assumed to have certain properties summarized in Assumption

1 in Section II.

Such unsupervised learning problems have been widely studied [1, 2]

. The problem is typically solved by applying clustering methods, e.g., k-means clustering

[3, 4, 5] and k-medoids clustering[6, 7, 8], where the data sequences are viewed as multivariate data with Euclidean distance as the distance metric. These clustering algorithms usually require the knowledge of the number of clusters and they differ in how the initial centers are determined. One reasonable way is to choose a data sequence as a center if it has the largest minimum distances to all the existing centers [9, 10, 11]. Alternatively, all the initial centers can be randomly chosen [12]. With the number of clusters unknown, there are typically two alternative approaches for clustering. One starts with a small number of clusters, e.g.,

1

, which is an underestimate of the true number, and proceed to split the existing clusters until convergence [13, 11]. The authors in [13] assumed a maximum number of clusters and the threshold for clustering depended on a pre-determined significance level of the two sample Kolmogorov-Smirnov (KS) test whereas the algorithm proposed in [11] did not assume a maximum number of clusters and the threshold for clustering was a function of the intra-cluster and inter-cluster distances. Alternatively, one may start with an overestimated the number of clusters, e.g., every sequence is treated as a cluster, and proceed to merge clusters that are deemed close to each other [11]. The algorithms in [9, 12, 13] were all validated by simulation results without carrying out an analysis of the error probability.

There are some key differences between the k-means algorithm and the k-medoids algorithm. The k-means algorithm minimizes a sum of squared Euclidean distances. Meanwhile, the k-medoids algorithm assigns data sequences as centers and minimizes a sum of arbitrary distances, which makes it more robust to outliers and noise

[14, 15]. Moreover, the k-means algorithm requires updating the distances between data sequences and the corresponding centroids in every iteration whereas the k-medoids algorithm only requires the pairwise distances of the data sequences, which can be computed before hand. Thus, the k-medoids algorithm outperforms the k-means algorithm in terms of computational complexity as the number of sequences increases [16].

Most prior research focused on computational complexity analysis, whereas the error probability and the performance comparison of different clustering algorithms were typically studied through simulations, e.g., [7, 17, 8, 16]

. This paper attempts to theoretically analyze the error probability for the k-medoids algorithm especially in the asymptotic region. Furthermore, in contrast to previous studies, which frequently used vector norms as the distance metric, e.g., Euclidean distance, our study adopts the distance metrics between distributions for clustering in order to capture the statistical models of data sequences considered in this paper. This formulation based on a distributional distance metric is uniquely suited to the proposed clustering problem, where each data point, i.e., each data sequence, represents a probability distribution and each cluster is a collection of closely related distributions, i.e., composite hypotheses.

Various distance metrics that take the distribution properties into account can be reasonable choices, e.g., KS distance [11, 10, 13, 12] and maximum mean discrepancy (MMD) [18]. Our previous work [11, 10] has shown that the KS distance based k-medoids algorithms are exponentially consistent for both known and unknown number of clusters. In this paper, we consider a much more general framework and instead of focusing on a specific distance metric such as KS distance in our prior work [11, 10], we consider any distance metric that satisfies Assumption 1 for clustering. The rationale is the following: with any distance metric that captures the statistical model of data sequences, one is likely to observe that for a large sample size, 1) sequences generated from the same distribution cluster are close to each other, and 2) sequences generated from different distribution clusters are far away from each other. Thus, if the minimum distance of different distribution clusters is greater than the maximum diameter of all the distribution clusters defined in Section II, then it becomes unlikely to make a clustering error as the sample size increases. In this paper, we develop k-mediods distribution clustering algorithms where the distances between distributions are selected to capture the underlying statistical model of the data. We analyze the error probability of the proposed algorithms, which takes the form of exponential decay as the sample size increases under Assumption 1. Furthermore, beyond the KS distance, we also consider another distance metric namely MMD and show that they both satisfy Assumption 1 so that the error probability of the proposed algorithms decays exponentially fast if the KS distance and MMD are used as the distance metrics.

We note that recent studies [19, 20, 21, 22, 23]

of anomaly detection problems and classification also took into account the statistical model of the data sequences. Our focus here is on the clustering problem, leading to error performance analysis that is substantially different from that in

[19, 20, 21, 22, 23].

The rest of the paper is organized as follows. In Section II, the system model of the clustering problem, the preliminaries of the KS distance and MMD, and notations are introduced. The clustering algorithm given the number of clusters and the corresponding upper bound on the error probability are provided in Section III, followed by the results of the clustering algorithms with an unknown number of clusters in Section IV. The simulation results for the KS and MMD based algorithms are provided in Section V.

Ii system model and preliminaries

Ii-a Clustering Problem

Suppose there are $K$ distribution clusters denoted by $P_{k}$ for $k = 1, \dots, K$ , where $K$ is fixed. Define the intra-cluster distance of $P_{k}$ and the inter-cluster distance between $P_{k}$ and $P_{k^{'}}$ for $k \neq k^{'}$ respectively as

	$d (P_{k})$	$= sup p_{i}, p_{i^{'}} \in P_{k} d (p_{i}, p_{i^{'}}),$		(1)
	$d (P_{k}, P_{k^{'}})$	$= inf p_{i} \in P_{k}, p_{i^{'}} \in P_{k^{'}} d (p_{i}, p_{i^{'}}),$		(1)

where $d (\cdot, \cdot)$ is a distance metric between distributions, e.g., the KS distance or MMD defined later in (5) and (6) respectively. Then $d (P_{k})$ and $d (P_{k}, P_{k^{'}})$ represent the diameter of $P_{k}$ and the distance between $P_{k}$ and $P_{k^{'}}$ , respectively. Define

$d_{L}$	$= max k = 1, \dots, K d (P_{k}),$	(2)
$d_{H}$	$= min k \neq k^{'} d (P_{k}, P_{k^{'}}),$
$Σ$	$= d_{H} + d_{L},$
$Δ$	$= d_{H} - d_{L} .$

Table I summarizes the notations of the generalized form of distances defined in (1) and (2) which will be used in the following discussion.

general	KS	MMD
$d (P_{k})$	$d_{K S} (P_{k})$	${M M D}^{2} (P_{k})$
$d (P_{k}, P_{k^{'}})$	$d_{K S} (P_{k}, P_{k^{'}})$	${M M D}^{2} (P_{k}, P_{k^{'}})$
$d_{L}$	$d_{L, k s}$	$d_{L, m m d}$
$d_{H}$	$d_{H, k s}$	$d_{H, m m d}$
$Δ$	$Δ_{k s}$	$Δ_{m m d}$
$Σ$	$Σ_{k s}$	$Σ_{m m d}$

TABLE I: Notations

Suppose $M_{k}$ data sequences are generated from the distributions in $P_{k}$ , and hence a total of $\sum_{k = 1}^{K} M_{k} = M$ sequences are to be clustered. Without loss of generality, assume that each sequence $x_{k, j_{k}}$ consists of $n$ independently identically distributed (i.i.d.) samples generated from $p_{k, j_{k}} \in P_{k}$ for $k = 1, \dots, K$ and $j_{k} \in {1, \dots, M_{k}}$ . The $i$ -th observation in $x_{k, j_{k}}$ is denoted by $x_{k, j_{k}} [i] \in R^{m}$ , where $m < \infty$ and $i \in {1, \dots, n}$ . Note that for any fixed $k$ , $p_{k, j_{k}}$ ’s are not necessarily distinct. Namely, for a fixed $k$ , some $x_{k, j_{k}}$ ’s can be generated from the same distribution. Although the following discussion assumes that all the data sequences have the same length, our analysis can be easily extended to the case with different sequence lengths by replacing $n$ with the minimum sequence length. In order to analyze the error probability of the clustering algorithm, we introduce an assumption relating to the concentration property of the distance metrics:

Assumption 1.

For any distribution clusters ${P_{1}, \dots, P_{K}}$ and any sequences $x_{k, j_{k}} \sim p_{k, j_{k}}$ , $x_{k, j_{k}^{'}} \sim p_{k, j_{k}^{'}}$ and $x_{k^{'}, j_{k^{'}}} \sim p_{k^{'}, j_{k^{'}}}$ , where $k \neq k^{'}$ , the following inequalities hold:


	$d_{L} < d_{H},$		(3a)
	$P (d (x_{k, j_{k}}, x_{k^{'}, j_{k^{'}}}) \leq d_{t h}) \leq a_{1} e^{- b n} \forall d_{t h} \in (d_{L}, d_{H}),$		(3b)
	$P (d (x_{k, j_{k}}, x_{k, j_{k}^{'}}) > d_{t h}) \leq a_{2} e^{- b n} \forall d_{t h} \in (d_{L}, d_{H}),$		(3c)
	$P (d (x_{k, j_{k}}, x_{k, j_{k}^{'}}) \geq d (x_{k, j_{k}}, x_{k^{'}, j_{k^{'}}})) \leq a_{3} e^{- b n},$		(3d)

where $a_{i}$ ’s are some constants independent of distributions, $b$ ( $> 0$ ) is a function of $d_{t h}$ and $n$ is the sample size. ∎

Here (3a) guarantees that the lower bound of inter-cluster distances is greater than the upper bound of intra-cluster distances. (3b) guarantees that the probability that the distance between two sequences generated from different distribution clusters is smaller than $d_{H}$ decays exponentially fast. (3c) guarantees that the probability that the distance between two sequences generated from the same distribution cluster is greater than $d_{L}$ decays exponentially fast. (3d) guarantees that given two sequences generated from the same cluster and a third sequence generated from another distribution cluster, the probability that the first sequence is actually closer to the third sequence decays exponentially fast.

A clustering output is said to be incorrect if and only if the sequences generated by different distribution clusters are assigned to the same cluster, or sequences generated by the same distribution cluster are assigned to more than one cluster. Denote by $P_{e}$ the error probability of a clustering algorithm. A clustering algorithm is said to be consistent if

lim n \to \infty P_{e} = 0,

where $n$ is the sample size. The algorithm is said to be exponentially consistent if

B = lim n \to \infty - \frac{1}{n} log P_{e} > 0.

For the case where a clustering algorithm is exponentially consistent, we are also interested in characterizing the error exponent $B$ .

Ii-B Preliminaries of KS distance

Suppose $x$ is generated by the distribution $p$ , where $x [i] \in R$

. Then the empirical cumulative distribution function (c.d.f.) induced by

x

is given by

F_{x} (a) = \frac{1}{n} n \sum i = 1 1_{[- \infty, a]} (x [i]),

(4)

where $1_{[- \infty, x]} (\cdot)$ is the indicator function. Let the c.d.f. of $p$ evaluated at $a$ be $F_{p} (a)$ . The KS distance is defined as

d_{K S} (p, q) = sup a \in R | F_{p} (a) - F_{q} (a) |,

(5)

where $F_{p} (a)$ and $F_{q} (a)$ can be either c.d.f’s of distributions or empirical c.d.f.’s induced by sequences.

Proposition 1.

The KS distance satisfies (3b) - (3d) if $d_{L, k s} < d_{H, k s}$ .

Proof.

See Lemmas .5, .3, and .7 in Appendix -A. ∎

Ii-C Preliminaries of MMD

Suppose $P$ includes a class of probability distributions, and suppose $H$ is the reproducing kernel Hilbert space (RKHS) associated with a kernel $g (\cdot, \cdot)$ . Define a mapping from $P$ to $H$ such that each distribution $p \in P$ is mapped into an element in $H$ as follows

μ_{p} (\cdot) = E_{p} [g (\cdot, x)] = \int g (\cdot, x) d p (x),

where $μ_{p} (\cdot)$ is referred to as the mean embedding of the distribution $p$ into the Hilbert space $H$ . Due to the reproducing property of $H$ , it is clear that $E_{p} [f] = ⟨ μ_{p}, f ⟩_{H}$ for all $f \in H$ .

It has been shown in [24, 25, 26, 27] that for many RKHSs such as those associated with Gaussian and Laplace kernels, the mean embedding is injective, i.e., each $p \in P$ is mapped to a unique element $μ_{p} \in H$

. In this way, many machine learning problems with unknown distributions can be solved by studying mean embeddings of probability distributions without actually estimating the distributions, e.g.,

[28, 29, 21, 22]. In order to distinguish between two distributions

p

and

q

, the authors in [30] introduced the following notion of MMD based on the mean embeddings

μ_{p}

and

μ_{q}

p

and

q

, respectively:

M M D (p, q) := ∥ μ_{p} - μ_{q} ∥_{H} .

(6)

An unbiased estimator of

{M M D}^{2} (p, q)

based on

x

and

y

which consist of

n

and

m

samples, respectively, is given by

		${M M D}^{2} (x, y)$		(7)
		$= \frac{1}{n (n - 1)} n \sum i = 1 n \sum j \neq i g (x [i], x [j]) + \frac{1}{m (m - 1)}$
		$m \sum i = 1 m \sum j \neq i g (y [i], y [j]) - \frac{2}{n m} n \sum i = 1 m \sum j = 1 g (x [i], y [j]) .$

Assume that the kernel is bounded, i.e., $0 \leq g (x, y) \leq K$ , where $K$ is finite.

Proposition 2.

The MMD distance satisfies (3b) - (3d) if $d_{L, m m d} < d_{H, m m d}$ .

Proof.

See Lemmas .6, .4 and .8 in Appendix -A. ∎

Ii-D Additional Notations

The following notations are used in the algorithms and the corresponding proofs. Let $C_{l}^{t}$ be the $l$ -th cluster obtained at the $t$ -th cluster update step and let $c_{l}^{t, a}$ , $c_{l}^{t, e}$ and $c_{l}^{t, s}$ be the centers of the $l$ -th cluster obtained by the center update step, merge step and split step of the $t$ -th iteration respectively for $t \geq 1$ . Moreover, let $C_{l}^{0}$ be the $l$ -th cluster obtained at the initialization step and $c_{l}^{0, a}$ be the corresponding center. Let ${^K}^{t}$ ( $t \geq 1$ ) be the number of centers before the $t$ -th cluster update step and the $t$ -th split step. Moreover, use ${^K}^{0}$ to denote the number of centers obtained at the center initialization step. For simplicity, all the superscripts are omitted in the following discussion when there is no ambiguity.

To further simplify the notation in the algorithms and the proofs, let ${y_{i}}_{i = 1}^{M}$ denote the data sequence set ${x_{k, j_{k}}}_{k = 1, j_{k} = 1}^{K, M_{k}}$ . However, the one-to-one mapping from ${y_{i}}_{i = 1}^{M}$ onto ${x_{k, j_{k}}}_{k = 1, j_{k} = 1}^{K, M_{k}}$ is not fixed, i.e., given a fixed $i$ , $y_{i}$ can be any sequence in ${x_{k, j_{k}}}_{k = 1, j_{k} = 1}^{K, M_{k}}$ unless other constraints are imposed. Denote by $y_{i} \sim P_{k}$ if $y_{i}$ is generated from a distribution $p \in P_{k}$ . Furthermore, define a set of integers

I_{k_{1}}^{k_{2}}

= {k_{1}, \dots, k_{2}},

where $k_{1}, k_{2} \in Z^{+}$ and $k_{1} < k_{2}$ .

Iii Known number of clusters

In this section, we study the clustering algorithm for known $K$ , the number of clusters. The method proposed in [9] is used for center initialization, as described in Algorithm 1. The initial $K$ centers are chosen sequentially such that the center of the $k$ -th cluster is the sequence that has the largest minimum distance to the previous $k - 1$ centers. The clustering algorithm itself is presented in Algorithm 2. Given the centers, each sequence is assigned to the cluster for which the sequence has the minimum distance to the center. For a given cluster, a sequence is assigned as the center subsequently if the sum of its distances to all the sequences in the cluster is the smallest. The algorithm continues until the clustering result converges.

1:Input: Data sequences

{y_{i}}_{i = 1}^{M}

, number of clusters

K

2:Output: Partitions

{C_{k}}_{k = 1}^{K}

3:{Center initialization}

4:Arbitrarily choose one

y_{i}

c_{1}

5:for

k = 2 to K

c_{k} \leftarrow arg {max}_{y_{i}} ({min}_{l \in I_{1}^{k - 1}} d (y_{i}, c_{l}))

7:end for

8:{Cluster initialization}

9:Set

C_{k} \leftarrow \emptyset

for

1 \leq k \leq K

10:for

j = 1 to M

11:

C_{l} \leftarrow C_{l} \cup {y_{i}}

, where

l = arg {min}_{l \in I_{1}^{K}} d (y_{i}, c_{l})

12:end for

13:Return

{C_{k}}_{k = 1}^{K}

Algorithm 1 Initialization with known

K

1:Input: Data sequences

{y_{i}}_{i = 1}^{M}

, number of clusters

K

2:Output: Partition set

{C_{k}}_{k = 1}^{K}

3:Initialize

{C_{k}}_{k = 1}^{K}

by Algorithm 1.

4:while not converge do

5: {Center update}

6: for

k = 1 to K

c_{k} \leftarrow arg {min}_{y_{i} \in C_{k}} \sum_{y_{j^{'}} \in C_{k}} d (y_{i}, y_{j^{'}})

8: end for

{Cluster update}

10: for

j = 1 to M

11: if

y_{i} \in C_{k^{'}}

and

d (y_{i}, c_{k}) < d (y_{i}, c_{k^{'}})

then

12:

C_{k} \leftarrow C_{k} \cup {y_{i}}

and

C_{k^{'}} \leftarrow C_{k^{'}} ∖ {y_{i}}

13: end if

14: end for

15:end while

16:Return

{C_{k}}_{k = 1}^{K}

Algorithm 2 Clustering with known

K

The following theorem provides the convergence guarantee for Algorithm 2 via an upper bound on the error probability.

Theorem III.1.

Algorithm 2 converges after a finite number of iterations. Moreover, under Assumption 1, the error probability of Algorithm 2 after $T$ iterations is upper bounded as follows

P_{e} \leq M^{2} (a_{1} + a_{2} + (T + 1) a_{3}) e^{- b n},

where $a_{1}$ , $a_{2}$ , $a_{3}$ and $b$ are as defined in Assumption 1.

Outline of the Proof.

The idea of proving the upper bound on the error probability is as follows. We first prove that the error probability at the initialization step decays exponentially. Note that the event that an error occurs during the first $T$ iterations is the union of the event that an error occurs at the $t$ -th step and the previous $t - 1$ iterations are correct for $t = 1, \dots, T$ . Thus, if we prove that the error probability at the $t$ -th step given correct updates from the previous iterations decays exponentially, then so does the error probability of the algorithm by the union bound argument. See Appendix -B1 for details. ∎

Theorem III.1 shows that for any given $K$ , any distance metric satisfying Assumption 1 yields an exponentially consistent k-medoids clustering algorithm with the error exponent $b$ .

Corollary III.1.1.

Suppose the KS distance and MMD are used for Algorithms 1 and 2, then

		$P_{e}^{K S} \leq M^{2} (6 T + 14) exp (- \frac{n Δ_{k s}^{2}}{8}),$
		$P_{e}^{M M D} \leq M^{2} (T + 3) exp (- \frac{n Δ_{m m d}^{2}}{256 K^{2}}) .$

Proof.

By Propositions 1 and 2, the upper bound on the error probability of Algorithm 2 applies to the KS distance and MMD. Thus, the corollary is obtained by substituting the values specified in Lemmas .3 - .8 in the upper bound. ∎

Corollary III.1.1 implies that Algorithm 2 is exponentially consistent under KS and MMD distance metrics with an error exponent no smaller than $\frac{Δ_{k s}^{2}}{8}$ and $\frac{Δ_{m m d}^{2}}{256 K^{2}}$ , respectively.

Iv Unknown number of clusters

In this section, we propose the merge- and split-based algorithms for estimating the number of clusters as well as grouping the sequences.

Iv-a Merge Step

If a distance metric satisfies (3c) and two sequences generated by distributions within the same cluster are assigned as centers, then, with high probability, the distance between the two centers is small. This is the premise of the clustering algorithm based on merging centers that are close to each other.

The proposed approach is summarized in Algorithms 3 and 4. There are two major differences between Algorithms 3 and 4 and Algorithms 1 and 2. First, the center initialization step of Algorithm 3 keeps generating an increasing number of centers until all the sequences are close to one of the existing centers. Second, an additional Merge Step in Algorithm 4 helps to combine clusters if the corresponding centers have small distances between each other.

1:Input: Data sequences

{y_{i}}_{i = 1}^{M}

and threshold

d_{t h}

2:Output: Partitions

{Ck}^Kk=1

{Center initialization}

4:Arbitrarily choose one

y_{i}

c_{1}

and set

^K = 1

5:while

{min}_{i \in I_{1}^{M}} ({min}_{k \in I_{1}^{^K}} d (y_{i}, c_{k})) > d_{t h}

c_{^K + 1} \leftarrow arg {max}_{y_{i}} ({min}_{k \in I_{1}^{^K}} d (y_{i}, c_{k}))

^K \leftarrow^K + 1

8:end while

9:Clustering initialization specified in Algorithm 1.

10:Return

{Ck}^Kk=1

Algorithm 3 Merge-based initialization with unknown

K

1:Input: Data sequences

{y_{i}}_{i = 1}^{M}

and threshold

d_{t h}

2:Output: Partition set

{Ck}^Kk=1

3:Initialize

{Ck}^Kk=1

by Algorithm 3.

4:while not converge do

5: Center update specified in Algorithm 2.

{Merge Step}

7: for

k1,k2∈{1,…,^K}

and

k_{1} \neq k_{2}

8: if

d (c_{k_{1}}, c_{k_{2}}) \leq d_{t h}

then

9: if

\sum_{y_{i} \in C_{k_{1}}} d (c_{k_{2}}, y_{i}) < \sum_{y_{i} \in C_{k_{2}}} d (c_{k_{1}}, y_{i})

then

10:

C_{k_{2}} \leftarrow C_{k_{1}} \cup C_{k_{2}}

and delete

c_{k_{1}}

and

C_{k_{1}}

11: else

12:

C_{k_{1}} \leftarrow C_{k_{1}} \cup C_{k_{2}}

and delete

c_{k_{2}}

and

C_{k_{2}}

13: end if

14:

^K \leftarrow^K - 1

15: end if

16: end for

17: Cluster update specified in Algorithm 2.

18:end while

19:Return

{Ck}^Kk=1

Algorithm 4 Merge-based clustering with unknown

K

Theorem IV.1.

Algorithm 4 converges after a finite number of iterations. Moreover, under Assumption 1, the error probability of Algorithm 4 after $T$ iterations is upper bounded as follows

P_{e} \leq M^{2} ((T + 1) a_{1} + a_{2} + (T + 1) a_{3}) e^{- b n},

where $a_{1}$ , $a_{2}$ , $a_{3}$ and $b$ are as defined in Assumption 1.

Proof.

The proof shares the same idea as that of Theorem III.1. See Appendix -B2 for details. ∎

Theorem IV.1 shows that the merge-based algorithm is exponentially consistent under any distance metric satisfying Assumption 1 with the error exponent $b$ .

Corollary IV.1.1.

Suppose the KS distance and MMD are used with $d_{t h} = \frac{Σ_{k s}}{2}$ and $d_{t h} = \frac{Σ_{m m d}}{2}$ . Then the error probability of Algorithm 4 after $T$ iterations is upper bounded as follows

		$P_{e}^{K S} \leq M^{2} (10 T + 14) exp (- \frac{n Δ_{k s}^{2}}{8}),$
		$P_{e}^{M M D} \leq M^{2} (2 T + 3) exp (- \frac{n Δ_{m m d}^{2}}{256 K^{2}}) .$

Proof.

By Propositions 1 and 2, the upper bound on the error probability of Algorithm 4 in IV.1 applies to the KS distance and MMD. Thus, the corollary is obtained by substituting the values specified in Lemmas .3 - .8 in the upper bound. ∎

Corollary IV.1.1 implies that Algorithm 4 is exponentially consistent under KS and MMD distance metrics with an error exponent no smaller than $\frac{Δ_{k s}^{2}}{8}$ and $\frac{Δ_{m m d}^{2}}{256 K^{2}}$ , respectively.

Iv-B Split Step

Suppose a cluster contains sequences generated by different distributions and the center is generated from $p \in P_{k}$ . Then if the distance metric satisfies (3b), the probability that the distances between sequences generated from distribution clusters other than $P_{k}$ and the center is small decays as the sample size increases. Therefore, it is reasonable to begin with one cluster and then split a cluster if there exists a sequence in the cluster that has a large distance to the center. The corresponding algorithm is summarized in Algorithm 5.

1:Input: Data sequences

{y_{i}}_{i = 1}^{M}

and threshold

d_{t h}

2:Output: Partition set

{Ck}^Kk=1

C_{1} = {y_{i}}_{i = 1}^{M}

^K = 1

and find

c_{1}

by center update specified in Algorithm 2.

4:while not converge do

5: {Split Step}

6: if

{max}_{k \in I_{1}^{^K}, y_{i} \in C_{k}} d (c_{k}, y_{i}) > d_{t h}

then

^K \leftarrow^K + 1

k = arg {max}_{k \in I_{1}^{^K}} ({max}_{y_{i} \in C_{k}} d (c_{k}, y_{i}))

c_{^K} \leftarrow arg {max}_{y_{i} \in C_{k}} d (c_{k}, y_{i})

10: end if

11: Cluster update specified in Algorithm 2.

12:end while

13:Return

{Ck}^Kk=1

Algorithm 5 Split-based clustering with unknown

K

Definition IV.1.1.

Suppose Algorithm 5 obtains $^K$ clusters at the $t$ -th iteration, where $^K < K$ and $^K = t$ or $t + 1$ . Then the correct clustering update result is that each cluster contains all the sequences generated from the distribution cluster that generates the center.

Theorem IV.2.

Algorithm 5 converges after a finite number of iterations. Moreover, under Assumption 1, the error probability of Algorithm 5 after $T$ iterations is upper bounded as follows

P_{e} \leq M^{2} T (a_{1} + a_{2} + a_{3}) e^{- b n},

where $a_{1}$ , $a_{2}$ , $a_{3}$ and $b$ are as defined in Assumption 1.

Outline of the Proof.

An error occurs at the $t$ -th iteration if and only if the $^K$ -th center is generated from distribution clusters that generated the previous centers or the clustering result is incorrect. Note that the error event of the first $T$ iterations is the union of the events that an error occurs at the $t$ -th iteration while the clustering results in the previous $t - 1$ iterations are correct for $t = 1, \dots, T$ . Similar to the proof of Theorem III.1, the error probability is bounded by the union bound. See Appendix -B3 for more details. ∎

Theorem IV.2 shows that the split-based algorithm is exponentially consistent under any distance metric satisfying Assumption 1 with the error exponent $b$ .

Corollary IV.2.1.

Suppose the KS distance and MMD are used with $d_{t h} = \frac{Σ_{k s}}{2}$ and $d_{t h} = \frac{Σ_{m m d}}{2}$ . Then the error probability of Algorithm 5 after $T$ iterations is upper bounded as follows

		$P_{e}^{K S} \leq 14 M^{2} T exp (- \frac{n Δ_{k s}^{2}}{8}),$
		$P_{e}^{M M D} \leq 3 M^{2} T exp (- \frac{n Δ_{m m d}^{2}}{256 K^{2}}) .$

Proof.

By Propositions 1 and 2, the upper bound on the error probability of Algorithm 5 in Theorem IV.2 applies to the KS distance and MMD. Thus, the corollary is obtained by substituting the values specified in Lemmas .3 - .8 in the upper bound. ∎

Corollary IV.2.1 implies that Algorithm 5 is exponentially consistent under KS and MMD with an error exponent no smaller than $\frac{Δ_{k s}^{2}}{8}$ and $\frac{Δ_{m m d}^{2}}{256 K^{2}}$ , respectively.

V Numerical Results

In this section, we provide some simulation results given $K = 5$ and $M_{k} = 3$ for $k = 1, \dots, 5$ . Moreover, $x_{k, j_{k}} [i] \in R$

. Gaussian distributions

N (μ_{k, j_{k}}, σ^{2})

and Gamma distributions

Γ (a_{k, j_{k}}, b)

are used in the simulations. The probability density function (p.d.f.) of

Γ (a, b)

is defined as

f (x; α, β)

= \frac{1}{β^{α} Γ (α)} x^{α - 1} exp (- \frac{x}{β}) (x > 0),

where $α > 0$ , $β > 0$ and $Γ (\cdot)$ is the Gamma function, respectively. Specifically, the parameters of the distributions are $μ_{k, j_{k}} \in {k - δ, k, k + δ}$ , $σ^{2} = 1$ , $α_{k, j_{k}} \in {2.5 k + 1 - δ, 2.5 k + 1, 2.5 k + 1 + δ}$ and $β = 1$ , where $δ = 0$ and $0.1$ . Note that when $δ = 0$ , all the sequences generated from the same distribution cluster are generated from a single distribution. The exponential kernel function is used in the simulations for the MMD distance, i.e.,

g (x, y) = e^{- \frac{| x - y |}{2}} .

(8)

V-a Known Number of Clusters

Simulation results for a known number of clusters are shown in Fig. 1. One can observe from the figures that by using both the KS distance and MMD, $log P_{e}$ is a linear function of the sample size, i.e., $P_{e}$ is exponentially consistent. Moreover, the logarithmic slope of $P_{e}$ with respect to $n$ , i.e., the quantity $- \frac{log P_{e}}{n}$ , increases as $δ$ becomes smaller, which, in the current simulation setting, implies a larger $Δ$ .

Furthermore, a good distance metric for Algorithm 2 depends on the underlying distributions. This is because the underlying distributions have different distances under the KS distance and MMD, which results in different values of $Δ_{k s}$ and $Δ_{m m d}$ .

V-B Unknown Number of Clusters

With an unknown number of distribution clusters, the threshold $d_{t h}$ specified in Corollaries IV.1.1 and IV.2.1 are used in the simulation. The performance of Algorithms 4 and 5 for the KS distance and MMD are shown in Figs. 2 and 3, respectively. Given the KS distance and MMD, $log P_{e}$ ’s are linear functions of the sample size when the sample size is large and larger $Δ$ implies a larger slope of $log P_{e}$ . Furthermore, given the same value of $δ$ , Algorithms 4 and 5 have similar performance under the KS distance whereas Algorithm 5 outperforms Algorithm 4 under MMD given $δ = 0$ and $0.1$ . Intuitively, smaller $δ$ implies larger $Δ$ in the current simulation setting, thereby should result in better clustering performance for a given sample size. However, Fig. 2LABEL:sub@fig:algorithm4_5_KS-b indicates that Algorithms 4 and 5 with KS distance performs better with $δ = 0.1$ than that with $δ = 0$ when the sample size is small. This is likely due to the fact that the KS distance between the two sequences is always lower bounded by $\frac{1}{n}$ . Thus, with small sample sizes, Algorithms 4 and 5 are likely to overestimate the number of clusters. This can be mitigated by the increased threshold $d_{t h}$ to control merging/splitting of cluster centers.

V-C Choice of $d_{t h}$

Note that in general $d_{t h} = ω d_{L} + (1 - ω) d_{H}$ , where $ω \in (0, 1)$ . Theorems IV.1 and IV.2 only establish the exponential consistency of Algorithms 4 and 5, respectively. We now investigate the impact on performance given different $ω$ ’s. One can observe from Fig. 4 that the choice of $d_{t h}$ has a significant impact on the performance of Algorithms 4 and 5. The optimal $d_{t h}$ depends on both the value of $δ$ and the underlying distributions. Moreover, from Fig. 4LABEL:sub@fig:algorithm4_5_alpha_KS-a-LABEL:sub@fig:algorithm4_5_alpha_KS-b, we can see that a smaller $ω$ which implies larger $d_{t h}$ results in better performance for KS distance and the two algorithms always have similar performance. On the other hand, from Fig. 4LABEL:sub@fig:algorithm4_5_alpha_MMD-a-LABEL:sub@fig:algorithm4_5_alpha_MMD-b, we can see that $ω = 0.5$ is a good choice for MMD if $P_{e}$ is of interest and (8) is used as the kernel function. Moreover, when $ω$ is small, Algorithm 5 outperforms Algorithm 4 under MMD.

Vi Conclusion

This paper studied the k-medoids algorithm for clustering data sequences generated from composite distributions. The convergence of the proposed algorithms and the upper bound on the error probability were analyzed for both known and unknown number of clusters. The error probability of the proposed algorithms were characterized to decay exponentially for distance metrics that satisfied certain properties. In particular, the KS distance and MMD were shown to satisfy the required condition, and hence the corresponding algorithms were exponentially consistent.

One possible generalization of the current work is to investigate the exponential consistency of other clustering algorithms given distributional distance metrics that satisfy the properties similar to that in Assumption 1.

-a Technical Lemmas

The following technical lemmas are used to prove Corollaries III.1.1, IV.1.1 and IV.2.1. All the data sequences in Lemmas .3 - .8 are assumed to consist of $n$ i.i.d. samples.

Lemma .1.

[Dvoretzky-Kiefer-Wolfowitz Inequality[31]] Suppose $x$ consists of $n$ i.i.d. samples generated from $p$ . Then

P (d_{K S} (x, p) > ϵ) \leq 2 exp (- 2 n ϵ^{2}) .

Lemma .2.

[McDiarmid’s Inequality[32]]

Consider independent random variables

x [1], \dots, x [n] \in X

and a mapping

f : X^{n} \to R

. If for all

i \in {1, \dots, n}

, and for any

~ x \in X

, there exist

c_{i} \leq \infty

for which

sup x \in X^{n}, ~ x \in X | f (x) - f (x^{'}) | \leq c_{i},

where

x^{'} [j] = {\begin{matrix} ~ x & if j = i, x [j] & otherwise, \end{matrix}

then for all probability measure $p$ and every $ϵ > 0$ ,

P (f (x) - E_{x} [f (x)] \geq ϵ) < exp (- \frac{2 ϵ^{2}}{\sum_{i = 1}^{n} c_{i}^{2}}),

where $E_{X} [\cdot]$ denotes the expectation over the $n$ random variables $x_{i} \sim p$ .

Lemmas .3 - .8 establish that the KS distance and MMD satisfy Assumption 1. Moreover, the lemmas provided in [10] are special cases of Lemmas .3, .5 and .7 with $d_{L} = 0$ .

Lemma .3.

Suppose $x_{j} \sim p_{j}$ for $j = 1, 2$ , where $p_{j} \in P$ and $d_{K S} (P) = d_{L, k s}$ . Then for any $d_{0} \in (d_{L, k s}, \infty)$ ,

P (d_{K S} (x_{1}, x_{2}) > d_{0}) \leq 4 exp (- \frac{n (d_{0} - d_{L, k s})^{2}}{2}) .

Proof.

Consider

		$P (d_{K S} (x_{1}, x_{2}) > d_{0})$
		$\leq P (d_{K S} (x_{1}, p_{1}) + d_{K S} (p_{1}, p_{2}) + d_{K S} (x_{2}, p_{2}) > d_{0})$
		$\leq P (d_{K S} (x_{1}, p_{1}) + d_{1} + d_{K S} (x_{2}, p_{2}) > d_{0})$

		$\leq 4 exp (- \frac{n {^d}^{2}}{2}),$

where $d_{L, k s} < d_{1} < d_{0}$ , $^d = d_{0} - d_{1}$ and ${lim}_{d_{1} ↓ d_{L, k s}} = d_{0} - d_{L, k s}$ . The first inequality is due to the triangle inequality of the $L_{1}$ -norm and the property of the supremum, and the last inequality is due to Lemma .1. Therefore, by the continuity of the exponential function, we have

P(dKS(x1,x2)>d0)≤4exp(−n(d0−dL,ks)22).\qed

Lemma .3 implies that the KS distance satisfies (3c) for $d_{t h} \in (d_{L, k s}, \infty)$ .

Lemma .4.

Suppose $x_{j} \sim p_{j}$ for $j = 1, 2$ , where $p_{j} \in P$ and ${M M D}^{2} (P) = d_{L, m m d}$ . Then for any $d_{0} \in (d_{L, m m d}, \infty)$ ,

P ({M M D}^{2} (x_{1}, x_{2}) > d_{0}) \leq exp (- \frac{n (d_{0} - d_{L, m m d})^{2}}{64 K^{2}}) .

Proof.

Without loss of generality, for any $γ \in {1, \dots, n}$ , consider $x_{2}^{'}$ , where

x_{2}^{'} [i] = {\begin{matrix} ~ x & if i = γ, x_{2} [i] & otherwise . \end{matrix}

Note that given $x_{1}$ , $x_{2}$ and $x_{2}^{'}$ ,

		$\| {M M D}^{2} (x_{1}, x_{2}) - {M M D}^{2} (x_{1}, x_{2}^{'}) ∣ ∣$

		$- \frac{2}{n^{2}} n \sum i = 1 (g (x_{2} [γ], x_{2} [i]) - g (~ x, x_{2} [i])) ∣ ∣ ∣ .$

Since $0 \leq g (x, y) \leq K$ , by the triangle inequality of $L_{1}$ -norm, we have

sup x_{1} \in X^{n}, x_{2} \in X^{n}, ~ x \in X ∣ ∣ {M M D}^{2} (x_{1}, x_{2}) - {M M D}^{2} (x_{1}, x_{2}^{'}) ∣ ∣ \leq \frac{8 K}{n},

where $X = R^{m}$ for any fixed $m$ . Moreover, notice that $E [{M M D}^{2} (x_{1}, x_{2})] = {M M D}^{2} (p_{1}, p_{2}) \leq d_{L, m m d}$ . Then

		$P ({M M D}^{2} (x_{1}, x_{2}) > d_{0})$
		$\leq P ({M M D}^{2} (x_{1}, x_{2}) - E [{M M D}^{2} (x_{1}, x_{2})] > d_{0} - d_{L, m m d})$
		$\leq exp (- \frac{n (d_{0} - d_{L, m m d})^{2}}{64 K^{2}}) .$

The last inequality is due to lemma .2. ∎

Lemma .4 implies that MMD satisfies (3c) for $d_{t h} \in (d_{L, m m d}, \infty)$ .

Lemma .5.

Suppose two distribution clusters $P_{1}$ and $P_{2}$ satisfy (3a) under the KS distance. Assume that for $j = 1, 2$ , $x_{j} \sim p_{j}$ where $p_{j} \in P_{j}$ . Then for any $d_{0} \in (0, d_{H, k s})$ ,

Proof.

Similar to the proof of lemma .3, we have

		$P (d_{K S} (x_{1}, x_{2}) \leq d_{0})$
		$\leq P (- d_{K S} (x_{1}, p_{1}) + d_{K S} (p_{1}, p_{2}) - d_{K S} (x_{2}, p_{2}) < d_{0})$
		$\leq P (- d_{K S} (x_{1}, p_{1}) + d_{2} - d_{K S} (x_{2}, p_{2}) < d_{0})$

		$\leq 4 exp (- \frac{n {^d}^{2}}{2}),$

where $d < d_{2} < d_{H}$ , $^d = d_{2} - d_{0}$ and ${lim}_{d_{2} ↑ d_{H, k s}} = d_{H, k s} - d_{0}$ . The last inequality is due to Lemma .1. Therefore, by the continuity of the exponential function, we have

Lemma .5 implies that the KS distance satisfies (3b) for $d_{t h} \in (0, d_{H, k s})$ .

Lemma .6.

Suppose two distribution clusters $P_{1}$ and $P_{2}$ satisfy (3a) under MMD. Assume that for $j = 1, 2$ , $x_{j} \sim p_{j}$ where $p_{j} \in P_{j}$ . Then for any $d_{0} \in (0, d_{H, m m d})$ ,

P ({M M D}^{2} (x_{1}, x_{2}) \leq d_{0}) \leq exp (- n (d_{H, m m d} - d_{0})^{2}

K-medoids Clustering of Data Sequences with Composite Distributions

Authors

Related Research

Nonparametric Composite Hypothesis Testing in an Asymptotic Regime

Nonparametric Detection of Anomalous Data Streams

The Exploitation of Distance Distributions for Clustering

Transductive-Inductive Cluster Approximation Via Multivariate Chebyshev Inequality

Interdependence of clusters measures and distance distribution in compact metric spaces

Distributed Detection with Empirically Observed Statistics

Non-Asymptotic Error Bounds for Bidirectional GANs

This week in AI

I Introduction

Ii system model and preliminaries

Ii-a Clustering Problem

Assumption 1.

Ii-B Preliminaries of KS distance

Proposition 1.

Proof.

Ii-C Preliminaries of MMD

Proposition 2.

Proof.

Ii-D Additional Notations

Iii Known number of clusters

Theorem III.1.

Outline of the Proof.

Corollary III.1.1.

Proof.

Iv Unknown number of clusters

Iv-a Merge Step

Theorem IV.1.

Proof.

Corollary IV.1.1.

Proof.

Iv-B Split Step

Definition IV.1.1.

Theorem IV.2.

Outline of the Proof.

Corollary IV.2.1.

Proof.

V Numerical Results

V-a Known Number of Clusters

V-B Unknown Number of Clusters

V-C Choice of dth

Vi Conclusion

-a Technical Lemmas

Lemma .1.

Lemma .2.

Lemma .3.

Proof.

Lemma .4.

Proof.

Lemma .5.

Proof.

Lemma .6.

V-C Choice of $d_{t h}$