Efficient Information Theoretic Clustering on Discrete Lattices

1 Motivation and Background

Clustering is an important tool of the trade in data mining and pattern recognition. Numerous algorithms have been proposed and an extended review of their characteristics and merits would be beyond the scope of this paper. However, we note that many algorithms used to cluster numerical data tacitly assume the data to be drawn from distributions that are continuous.

Alas, many data are explicitly discrete. Consider, for instance, the pixels of a standard digital image. Owing to modern recording technologies, domain and range of such images are discrete and we may distinguish three elementary cases:

(i) In a binary (black and white) image, each pixel is switched either on or off. Pixels that are switched off need not be represented; pixels that are switched on are characterized by their two-dimensional grid coordinate $[u, v]^{T}$ .

(ii) In an intensity (gray value) image, pixels are located on a two-dimensional grid and assume discrete gray values; they are thus fully characterized by a three-dimensional grid coordinate $[u, v, g]^{T}$ .

(iii) In a color (RGB) image, each pixel has a fixed location on a two-dimensional grid and maps to a discrete three-dimensional color vector; color pixels are thus fully characterized by a five-dimensional grid coordinate

[u, v, r, g, b]^{T}

Digital image therefore correspond to sets of data points that reside on low dimensional lattices. If such data need to be clustered into coherent groups, it is common practice to resort to algorithms that perform continuous optimization and thus will likely produce cluster centers that are located in between lattice points. In most practical applications, this is perfectly acceptable and our concern in this paper is therefore not with precision but with efficiency.

In this paper, we adapt an algorithm for continuous, information theoretic clustering to data on discrete lattices. Our basic idea is to connect recent machine learning techniques with well established methods for digital signal processing.

We show that recasting essential steps of information theoretic clustering in terms of efficient convolution operations significantly reduces its runtime and enables us to quickly compute cluster centers in image data. To simplify our discussion, we focus on applications in binary image processing and provide qualitative examples to illustrate behavior and performance of the algorithm. Quantitative experiments on a large data base of binary images reveal that our accelerated algorithm is two orders of magnitude faster than the original one. Moreover, we demonstrate that weighted clustering is straightforward in our framework.

Next, we review information theoretic clustering and discuss its properties. Then, we derive our accelerated version and present experimental results that underline its favorable performance. Finally, we discuss how the proposed algorithm immediately allows for efficient weighted clustering and conclude this paper with a summary and an outlook to practical applications of our approach.

2 Information Theoretic Clustering

The particular approach to information theoretic clustering (ITC) we consider in this paper was introduced in [11, 12, 13]. Given a set

X = {x_{1}, \dots, x_{N}}, x_{i} \in R^{d}

of data points drawn from a continuous domain, it applies an entropy-based measure to determine $M ≪ N$ clusters in the data. Similar to $k$ -means clustering, ITC starts from a random initialization of the cluster centers

W = {w_{1}, \dots, w_{M}}, w_{j} \in R^{d}

and iteratively updates these codebook vectors in order to minimize the entropy measure under consideration.

2.1 Iterative Entropy Minimization

ITC assumes that two probability density functions $p (x)$ and $q (x)$ are available that characterize the sets $X$ and $W$ , respectively. Given these densities, ITC optimizes the placement of codebook vectors by means of minimizing the Cauchy-Schwartz divergence

D_{c s} (X, W) = 2 H (X; W) - H (W) - H (X)

(1)

between $p (x)$ and $q (x)$ . Note that the last term of $D_{c s}$ does not depend on $W$ . The entropies in its first and second term are given by

	$H (X; W)$	$= - log \int p (x) q (x) d x! = - log V (X; W)$
and
	$H (W)$	$= - log \int q^{2} (x) d x! = - log V (W)$

and are known as Renyi’s cross entropy between $p (x)$ and $q (x)$ and Renyi’s entropy of $q (x)$ , respectively [14].

The fundamental idea in [11, 12, 13] is to model the densities $p (x)$ and $q (x)$ using their Parzen estimates

	$p (x)$	$= \frac{1}{N} N \sum i = 1 G_{ξ} (x - x_{i})$		(2)
	$q (x)$	$= \frac{1}{M} M \sum j = 1 G_{ω} (x - w_{j})$		(3)

where $G_{ξ} (x)$ and $G_{ω} (x)$

are Gaussian kernels of variance

ξ^{2}

and

ω^{2}

, respectively.

Applying the Gaussian product theorem and using properties of the so called overlap integral, $V (X; W)$ and $V (W)$ can be written as

	$\int p (x) q (x) d x$	$= \frac{\sum_{i, j} \int G_{ξ} (x - x_{i}) G_{ω} (x - w_{j}) d x}{M N}$
		$= \frac{1}{M N} N \sum i = 1 M \sum j = 1 G_{τ} (x_{i} - w_{j})$
and
	$\int q^{2} (x) d x$	$= \frac{\sum_{i, j} \int G_{ω} (x - w_{i}) G_{ω} (x - w_{j}) d x}{M^{2}}$
		$= \frac{1}{M^{2}} M \sum i = 1 M \sum j = 1 G_{ρ} (w_{i} - w_{j})$

where $τ^{2} = ξ^{2} + ω^{2}$ and $ρ^{2} = 2 ω^{2}$ .

Therefore, differentiating $D_{c s} (X, W)$ with respect to $w_{k}$ , equating to zero, and rearranging the resulting terms yields the following fix point update rule for every codebook vector in $W$

$w_{k}^{new} =$	$\frac{\sum_{j = 1}^{N} G_{τ} (x_{j} - w_{k}) x_{j}}{\sum_{j = 1}^{N} G_{τ} (x_{j} - w_{k})}$
	$- c \cdot \frac{\sum_{j = 1}^{M} G_{ρ} (w_{j} - w_{k}) w_{j}}{\sum_{j = 1}^{N} G_{τ} (x_{j} - w_{k})}$
	$+ c \cdot \frac{\sum_{j = 1}^{M} G_{ρ} (w_{j} - w_{k})}{\sum_{j = 1}^{N} G_{τ} (x_{j} - w_{k})} w_{k}$	(4)

where the constant $c = \frac{N}{M} \frac{V (X; W)}{V (W)}$ .

2.2 Properties of ITC

In [12], it is shown that the popular mean shift procedure [7, 8] is a special case of ITC. The cross entropy term $log V (X; W)$ in the Cauchy-Schwartz divergence causes ITC to be a mode seeking algorithm. The quantity $log V (W)$ , however, can be understood as the potential energy of a repellent force between the codebook vectors [12]. After convergence of the algorithm, the codebook vectors will therefore be located close to local modes but because of the repellent force they will not have collapsed into only a few modes.

Figure 1 illustrates how ITC determines a codebook of 30 vectors from a binary shape image. Every foreground pixel of the original image in the leftmost panel is understood as a 2D vector $x_{i} \in R^{2}$ ; together they form the data set $X$ . Using (2) to compute an estimate of the density $p (x)$ of $X$ yields the result shown in the second panel. The third panel demonstrates that a random initialization of a set of codebook vectors $W$ typically yields a density function $q (x)$ that diverges from the density $p (x)$ of the data points. Iterative updates of the codebook vectors using (4) will minimize this divergence. The rightmost panel of the figure shows how, upon termination of the ITC algorithm, the elements in $W$ have moved towards local modes of $X$ and that the two densities $p (x)$ and $q (x)$ are now in close agreement. Since the approach prevents codebook vectors from collapsing into a few modes, the final distribution $q (x)$ is found to trace out local principal curves of the binary shape.

For $N$ data points and $M$ codebook vectors, every iteration of ITC requires efforts of $O (M N)$ . However, in contrast to related clustering algorithms of similar complexity such, for instance, $k$ -means, there is a large constant that factors into the overall runtime of ITC. Each of the $k = 1, \dots, M$ updates in (4) requires the computation of $N$ distances $∥ x_{j} - w_{k} ∥^{2}$ and $M$ distances $∥ w_{j} - w_{k} ∥^{2}$ and each distance computation entails the evaluation of an exponential function in a Gaussian kernel.

With respect to modern practical applications, for instance in large scale image processing, the algorithm will therefore be too slow to be useful. It would neither allow for an efficient processing of a few very large images nor would it apply to the processing of large amounts of simple binary images of only moderate size.

3 ITC on Discrete Lattices

Addressing the efficiency concerns raised in the previous section, we now derive an alternative update rule for information theoretic clustering. Our key contribution at this point is to model the Parzen density estimates in (2) and (3) using methods from signal processing. Mathematically, our result is equivalent to the original formulation but it is cast in a form that allows for avoiding repeated computations of distances and exponentials. Especially for data that reside on discrete lattices our new formulation of ITC enables highly efficient implementations based on discrete filter masks and therefore considerably accelerate the procedure. To ease our discussion, we will again focus on examples from binary image processing but the methods discussed below immediately generalize to higher dimensional lattices, too.

3.1 Entropy Minimization Using Convolutions

Recall that the convolution of two functions $f (x)$ and $g (x)$ is defined as

(f * g) (x) = \int f (y) g (x - y) d y

where integration is over the whole, possibly higher dimensional domain.

To begin our derivation, we note that the convolution of a function $f$ with a shifted delta impulse $δ (x - μ)$ shifts the function

δ (x - μ) * f (x) = f (x - μ) .

A Parzen density estimate using a sum of Gaussians each centered at data points $x_{i}$ is therefore equivalent to a sum of shifted delta impulses convolved with a Gaussian

\sum i G_{σ} (x - x_{i}) = G_{σ} (x) * \sum j δ (x - x_{i}) .

With respect to applications in binary image processing, this observation is pivotal because every active binary pixel can be understood as a discrete delta impulse located at some coordinate on a regular 2D lattice. In a slight abuse of notation, we may therefore write the density function of a set $X$ of pixels in (2) as

p (x) = X * G_{ξ} .

Convolution with a Gaussian is a standard and well understood operation in low-level signal processing. In digital image processing, it typically realized using discrete 2D filter masks. In cases where $ξ$ is small, the Gaussian convolution kernel can be separated into two 1D convolution kernels. Successively applying corresponding pre-computed 1D filters of small spatial support will then significantly reduce computation time. In cases where $ξ$ is large, it is more efficient to resort to multiplication in the Fourier domain or to apply recursive filters [9, 17]. The density $p (x)$ of a set of binary pixels can therefore be computed very efficiently; although it will still require efforts proportional to the number $N$ of pixels, the overall effort per pixel will become very small.

Note that the set of codebook vectors $W$ , too, can be understood as a collection of delta impulses on a discrete lattice. With respect to binary image processing, this is to say that $W$ can be understood as an image containing $M$ foreground pixels. Accordingly, its density estimate

q (x) = W * G_{ω}

can be computed just as efficiently. On a discrete lattice such as an image pixel array, the two entropy integrals $V (X; W) = \int p (x) q (x) d x$ and $V (W) = \int q^{2} (x) d x$ are therefore readily available.

It thus remains to derive a modified update rule for the codebook vectors that assumes the role of (4). To this end, we have another look at (3) and note that

	$\frac{\partial q (x)}{\partial w_{k}}$	$= \frac{\partial}{\partial w_{k}} \frac{1}{M} M \sum j = 1 G_{ω} (x - w_{j})$
		$= \frac{1}{M} \frac{w_{k} - x}{2 ω^{2}} G_{ω} (x - w_{k}) .$		(5)

Using this, we differentiate the divergence $D_{c s} (X, W)$ in (1) with respect to $w_{k}$ , equate to zero, rearrange the resulting terms, and obtain the following update rule:

$w_{k}^{new} =$	$\frac{\int p (x) G_{ω} (x - w_{k}) x d x}{\int p (x) G_{ω} (x - w_{k}) d x}$
	$- c \cdot \frac{\int q (x) G_{ω} (x - w_{k}) x d x}{\int p (x) G_{ω} (x - w_{k}) d x}$
	$+ c \cdot \frac{\int q (x) G_{ω} (x - w_{k}) d x}{\int p (x) G_{ω} (x - w_{k}) d x} w_{k}$	(6)

where $c = \frac{V (X; W)}{V (W)}$ .

Note, that in contrast to (4), every Gaussian kernel in (6) is of the form $G_{ω} (x - w_{k}) = δ (x - w_{k}) * G_{ω} (x)$ . That is, they are now centered at $w_{k}$ , are all of the same variance, and do not specifically depend on the $x_{i}$ or the other $w_{j}$ anymore.

On a discrete lattice, this allows for further acceleration. Following common practice in digital signal processing, we can approximate the continuous Gaussian $G_{ω} (x)$ by means of a pre-computed discrete filter mask $F$ of finite support. Instead of evaluating the integrals in (6) over the whole lattice, it then suffices to consider a neighborhood $N$ of $w_{k}$ whose size depends on $ω$ . That is, on a discrete lattice, it is valid to assume the approximation

\int p (x) G_{ω} (x - w_{k}) x d x

\approx \sum x \in N p (x) F (x - w_{k}) x

and similar expressions hold for the other integrals in (6).

Concluding our derivation up to this point, we observe that, in contrast to the original update algorithm for ITC, our version for discrete lattices (i) avoids the computation of Euclidean distances and (ii) does not necessitate explicit evaluation of exponentials. On a discrete lattice both these steps can be replaced by highly efficient convolution operations using pre-computed filter masks.

3.2 ITC for Binary Images

In this subsection, we discuss implementation details of our accelerated ITC algorithm. To simplify our discussion, we focus once again on binary images and show how they may be processed efficiently.

For binary image processing, our efficient version of the original ITC algorithm draws on an isomorphism between a binary image matrix $X = [X_{u v}]$ and a finite set $X$ of vectors $x_{i} \in N^{2}$ . That is, in an implementation of our algorithm as well as in our discussion, we capitalize on the isomorphism

X_{u v} = {\begin{matrix} 1, & if x_{i} = [u, v]^{T} \in X 0, & otherwise \end{matrix}

so that we may use $X$ and $X$ interchangeably. Of course, a similar equivalence also holds for the set of codebook vectors so that we may refer to them using either $W$ or $W$ .

1: randomly initialize

W

, for instance by sampling from the foreground pixels of

X

2: compute the pdf

p (x)

using

P = X * G_{ξ}

3: repeat

4: compute the pdf

q (x)

using

Q = W * G_{ω}

5: compute the entropy

V (X; W) = \sum_{u, v} P_{u v} Q_{u v}

6: compute the entropy

V (W) = \sum_{u v} Q_{u v}^{2}

7: for

k = 1, \dots, M

8: compute

w_{k}^{new}

using (6)

9: end for

10: update

W = ⋃_{k} w_{k}^{new}

11: until

D_{c s} (X; W) \leq ϵ

Algorithm 1 Efficient ITC for binary image quantization

Given these prerequisites, Algorithm 1 summarizes how to use ITC in order to cluster binary images into coherent segments. The procedure begins with randomly sampling $M$ codebook vectors $w_{j}$ from the pixels $x_{i}$ of the shape image $X$ . Discrete approximations $P$ and $Q$ of the two probability density functions $p (x)$ and $q (x)$ are (pre-)computed using convolutions.

Also, since the algorithm is tailored to data on discrete lattices, the integrals in $V (X; W)$ and $V (W)$ are replaced by finite sums over the elements of $P$ and $Q$ . With these ingredients, the codebook vectors $w_{k}^{new}$ are updated according to equation (6). These steps are iterated until the Cauchy-Schwartz divergence between $P$ and $Q$ falls below a user defined threshold or the updates become very small. In practice, we found that it typically requires less than 20 iterations for the procedure to converges to a stable solution.

4 Performance Evaluation

Next, we present and discuss results obtained from experimenting with our accelerated version of information theoretic clustering and the original algorithm according to [11, 12, 13]. In addition, we present results obtained from applying $k$ -means clustering to the pixels of binary images in order to provide a well known baseline for comparison.

Figure 2 provides an example of the placement of $5$ as well as of $30$ codebook vectors on a binary shape. The results shown here are representative for what we found in our experiments. The original version of ITC and our adapted algorithm produce almost identical results. This was to be expected, since mathematically the two are equivalent. The minor differences visible in the figure can be attributed to the fact, that the original ITC algorithm performs a continuous optimization, while our version places the codebook vectors on a discrete grid of coordinates using filter masks of finite support.

Interestingly, for a larger number of clusters the codebook obtained from $k$ -means clustering is rather similar to the ones obtained from ITC. However, $k$ -means clustering considers variance minimization for optimization and places cluster centers at local means. It therefore tends to produce blob-like clusters of comparable sizes with equidistant centers. This becomes especially apparent, if a smaller number of clusters are to be determined. In such cases, ITC yields results that are noticeably different. This difference is further stressed in Fig. 3 which, in addition to the the different cluster centers produced by $k$ -means and ITC, also shows the corresponding segmentations of the shape.

Figure 4: Left panel: average computing time per iteration w.r.t. shape size. The plots indicate how accelerated ITC, original ITC, and $k$
-means perform in extracting 100 codebook vectors from shapes of growing size. Right panel: average computing time per iteration w.r.t. number of codebook vectors extracted from a shape of about 10.000 pixels. Original ITC and k-means show linear growth (note the logarithmic scale); for accelerated ITC, our way of choosing local support regions (see text) even decreases runtime.

Figure 4 summarizes quantitative results obtained from runtime experiments with the MPEG-7 shape data set. All our experiments on runtime behavior of the different algorithms were carried out on an Intel Core i3 processor. The variance parameter $ω$ in our accelerated ITC algorithm was set dynamically, taking into account the number $N$ of pixels in a shape and the number $M$ of codebook vectors to be produced. By choosing $ω = \sqrt{N / M} / 2$ , a smaller codebook will lead to larger discrete filter masks applied in computing ITC according to (6). Note that in order to benefit from the idea of using localized filter masks of finite support, it may in fact be preferable to consider convolution masks of smaller diameter. The parameter $ξ$ was set to $ω / 2$ . Finally, all the results we present here were obtained form averaging over 10 trials with different random initializations of $W$ .

Both, the left and right panel of Figures 4 underline the runtime improvement of accelerated ITC over the original algorithm. For larger choices of $N$ and $M$ , the time per iteration of our accelerated algorithm is two orders of magnitude smaller than the time required by the original version. The declining time per iteration of accelerated ITC for a growing number of clusters is a consequence of the above choice of $ω$ and its impact on the size of filter masks.

5 Weighted ITC

In addition to its favorable runtime characteristics, our adapted ITC algorithm also easily extends towards efficient weighted clustering. Our contribution in this section is to replace the simple Parzen estimate of the density $p (x)$ of the data $X$

by a more flexible Gaussian mixture model. As we shall see next, for data on a discrete lattice this does not cause any noticeable overhead for our algorithm.

If we extend the model for $p (x)$ in equation (2) to the more general form

p (x) = N \sum i = 1 h (x_{i}) G_{ξ} (x - x_{i}),

we immediately recognize yet another convolution

p (x) = (h * G_{ξ}) (x) .

We also note that in our adapted version of the ITC algorithm there is only one place where $p (x)$ is computed explicitly. This happens in the second preparatory step just outside of the loop that iteratively updates the codebook vectors.

With respect to the quantization of binary images, we can therefore introduce a weighting scheme for the pixels in $X$ at almost no costs. For instance, we may apply efficient distance transforms [6] to the foreground pixels of $X$ to assign higher weights to the pixels in the interiors of a shape. If we denote this transform by $T (X)$ , the second step in our algorithm simply becomes $P = T (X) * G_{ξ}$ .

Figures 5 and 6 show examples of the effect of assigning higher weights to pixels in the interior of a shape. In contrast to the normal variant of ITC, the weighted version causes the cluster centers to be concentrated closer to the shape interiors. Since the weighting scheme only impacts the mode seeking component of the algorithm but not the term that causes cluster centers to repel each other, the resulting codebook vectors trace local principal curves of the shape considered here.

Figure 6: . Examples of clusters or shape segments after computing $k = 5$ codebook vectors using weighted ITC based on the distance transform illustrated in Fig. 5.

6 Summary and Outlook

This paper introduced an efficient algorithm for clustering data that reside on a discrete lattice. We demonstrated how costly operations in information theoretic clustering can be be replaced by convolutions. Since convolutions on low dimensional lattices can be computed efficiently, our modified algorithm provides a powerful tool for the quantization of image pixels or voxels.

We illustrated the behavior of the algorithm by means of binary shape images. In experiments carried out to assess the runtime behavior of the modified procedure, we found it to be two orders of magnitude faster than the original algorithm but nevertheless to yield results of similar quality.

Finally, we demonstrated that for data on a lattice, it is straightforward to introduce weighting schemes into the adapted algorithm. Again, given the example of binary images, we showed the effect of applying distance transforms for the purpose of weighting. With this adaptation, the cluster centers produced by the algorithm trace local principal curves or latent features of a shape.

Binary images are a straightforward application domain for our algorithm because their pixels form a distribution of 2D data points. Gray value- and color images, too, can be understood as distributions on low-dimensional lattices. Yet, in practice it would introduce considerable memory overhead to represent them this way. At first sight, the practical applicability of the method proposed in this paper may therefore seem limited. However, this is not the case! On the contrary, modern imaging technology such as tomography or hyper-spectral imaging produces data that meet the prerequisites for our method.

Hyper-spectral images record a whole spectrum of wavelengths for every spatial coordinate of an image. They can thus be thought of as three-dimensional lattices where the data at each grid point is physically related to its neighbors. In this sense, a hyper-spectral image forms a discrete three-dimensional distribution of measurements and accelerated ITC provides a useful tool to uncover latent structures in these data.

In fact, due to their enormous data volumes, hyper-spectral images pose considerable challenges for statistical analysis as traditional methods do simply not apply anymore [10, 16]. Yet, there is a lot to gain. For example, in areas such as biology or phytomedicine, hyper-spectral data has been shown to provide new insights into physiological processes within plants [1, 2, 3] and using accelerated ICT promises further interesting results in this demanding big data domain.

Another interesting, though lesser known application area, lies in the field of game AI [4, 5, 15]. Many modern computer games that are set in simulated 3D worlds require the player to act according to his or her current location. For the programming of convincing artificial game agents this poses the problem of equipping them with a sense of where they are. However, due to the growing size and complexity of virtual, manual annotations that capture all essential areas and aspects may soon become infeasible. Accelerated ICT therefore provides a data-driven approach to the semantic clustering of in-game navigation meshes and first results already underline that it indeed uncovers key locations of in-game maps that make sense from a human player’s point of view.

It is in areas like these, where the method introduced in this paper is of considerable practical benefit and where bridges between advanced machine learning and highly efficient signal processing are in dire need.

References

[1] A. Ballvora, C. Römer, M. Wahabzada, U. Rascher, C. Thurau, C. Bauckhage, K. Kersting, L. Plümer, and J. Leon. Deep Phenotyping of Early Plant Response to Abiotic Stress Using Non-invasive Approaches in Barley. In Proc. Int. Barley Genetics Symposium, 2012.
[2] C. Bauckhage and K. Kersting. Data Mining and Pattern Recognition in Agriculture. KI – Künstliche Intelligenz, 27(4):313–324, 2013.
[3] C. Bauckhage, K. Kersting, and A. Schmidt. Agriculture’s Technological Makeover. IEEE Pervasive Computing, 11(2):4–7, 2012.
[4] C. Bauckhage, M. Roth, and V. Hafner.
Where am I? – On Providing Gamebots with a Sense of Location Using Spectral Clustering of Waypoints.
In Proc. Optimizing Player Satisfaction in Computer and Physical Games, 2006.
[5] C. Bauckhage and C. Thurau. Towards a Fair ’n Square Aimbot – Using Mixtures of Experts to Learn Context Aware Weapon Handling. In Proc. GAME-ON, 2004.
[6] G. Borgefors. An Improved Version of the Chamfer Matching Algorithm. In Proc. ICPR, 1984.
[7] Y. Cheng. Mean Shift, Mode Seeking, and Clustering. IEEE Trans. PAMI, 17(8), 1995.
[8] D. Comaniciu and P. Meer. Mean Shift: A Robust Approach towards Feature Space Analysis. IEEE Trans. PAMI, 24(5), 2002.
[9] R. Deriche. Recursively Implementing the Gaussian and its Derivatives. In Proc. ICIP, 1992.
[10] K.Kersting, Z. Xu, M. Wahabzada, C. Bauckhage, C. Thurau, C. Römer, A. Ballvora, U. Rascher, J. Leon, and L. Plümer. Pre-symptomatic Prediction of Plant Drought Stress using Dirichlet-Aggregation Regression on Hyperspectral Images. In Proc. AAAI, 2012.
[11] T. Lehn-Schioler, A. Hedge, D. Erdogmus, and J. Principe. Vector Quantization Using Information Theoretic Concepts. Natural Computing, 4(1), 2005.
[12] S. Rao, A. de Medeiros Martins, and J. Principe. Mean Shift: An Information Theoretic Perspective. Pattern Recognition Letters, 30(3), 2009.
[13] S. Rao, S. Han, and J. Principe. Information Theoretic Vector Quantization with Fixed Point Updates. In Proc. IJCNN, 2007.
[14] A. Renyi. On Measures of Information and Entropy. In Proc. Berkeley Symp. on Mathematics, Statistics, and Probability, 1961.
[15] R. Sifa and C. Bauckhage. Archetypical Motion: Supervised Game Behavior Learning with Archetypal Analysis. In Proc. CIG, 2013.
[16] C. Thurau, K. Kersting, M. Wahabzada, and C. Bauckhage. Descriptive Matrix Factorization for Sustainability: Adopting the Principle of Opposites. Data Mining and Knowledge Discovery, 24(2):325–354, 2012.
[17] L. van Vliet, I. Young, and P. Verbeek. Recursive Gaussian Derivative Filters. In Proc. ICPR, 1998.

Efficient Information Theoretic Clustering on Discrete Lattices

Undecidability of Underfitting in Learning Algorithms

An Information-Theoretic Perspective on Overfitting and Underfitting

Bayesian estimation of information-theoretic metrics for sparsely sampled distributions

Information-Theoretic Analysis of Minimax Excess Risk

On the optimality of kernels for high-dimensional clustering

Co-Clustering via Information-Theoretic Markov Aggregation

Nonparametric Extrema Analysis in Time Series for Envelope Extraction, Peak Detection and Clustering

1 Motivation and Background