During the past decades, a large number of theoretical results have been obtained for supervised learning such as classification and regression. For unsupervised learning, however, relatively few theoretical results are available. A main difficulty is that the objective functions in unsupervised learning are usually non-convex and multi-modal, so the optimization algorithms usually cannot find the global optima. As a result, it is generally difficult to obtain theoretical guarantees for the performances of the unsupervised learning algorithms. A simple and typical example of unsupervised learning is clustering or learning mixture models, and a typical algorithm for fitting the mixture models is the EM algorithm 
, which is a statistical counterpart of the k-means algorithm for clustering. Although the EM algorithm is simple and interpretable, and is known to converge monotonically to a local mode of the observed-data log-likelihood, little is known about its theoretical performance in terms of correctly recovering the mixture components. As such, the EM algorithm is often considered a heuristic algorithm.
A major recent advance in the theoretical understanding of the EM algorithm for fitting mixture models was made by Dasgupta and Shulman 
. They proposed a two-round variant of the EM algorithm that consists of only two iterations of EM: the first iteration is initialized from a number of randomly selected training examples as the centers of the Gaussian distributions, and the second iteration is carried out after pruning the clusters learned from the first iteration. They showed that the two-round EM can learn the mixture of Gaussian distributions with near optimal precision with high probability if the Gaussian distributions are well separated and if the dimensionality of the Gaussian distributions is sufficiently high. Here near optimal precision means that one can estimate the parameters of the Gaussian distributions as if the memberships of the observations are known.
In this paper, we generalize the theory of Dasgupta and Shulman  to learning mixture of Bernoulli templates. Each template is a binary vector, and it generates examples by independently switching its binary components with a certain probability. So the observed examples are also binary vectors. This setup is a version of the latent class model of  restricted to binary data. In potential applications in computer vision, a binary vector is a feature map of an image, where each binary component indicates whether a local feature or structure is present or absent within a certain cell of the image domain. Fig. 1 illustrates the basic idea by a synthetic example. The image domain is equally partitioned into squared cells (in the example in Fig. 1, there are a total of cells in the image domain). There is an alphabet of sketch patterns that can appear in these cells (Fig. 1 shows an alphabet of 18 types of sketch patterns). Each cell may contain one or more sketch patterns, so the binary vector for each image consists of binary components, each component indicates whether a certain sketch pattern is present or not within a certain cell. Specifically, each component is a binary decision that can be made based on local edge detection, Gabor filter responses , beamlet transformation 
or a pre-trained classifier. A Gabor filter is a 2D linear filter that has a prefered orientation. Along that orientation the Gabor filter resembles a Gaussian and along the perpendicular direction it resembles the derivative of a Gaussian. It was shown in
that the Gabor filters are a good approximation of the receptive field profiles of orientation-sensitive neurons in a cat’s visual cortex.
The formulation is very general. One can design any alphabet of local features or patterns, and one can use any binary detector or classifier to decide the presence or absence of these features within each cell. The whole feature map is a composition of local image features and is in the form of a binary vector, usually high dimensional (on the order of ). A template itself is a binary vector that is subject to component-wise switching or Bernoulli noise to account for the variations of the feature maps of individual images. The reason we focus on binary feature maps in this article is that they are easy to design and we do not need to make strong assumptions on their distributions such as Gaussianity.
As another illustration, Fig. 2 displays some examples of real images and their binary sketches based on a simple design of image features and binary decision rules. We partition the image domain into squared cells of equal size (in these images, the cells are relatively small, ranging from pixels to pixels). We convolve the image with Gabor filters  at 8 orientations. Within each cell, at each orientation, we pool a local maximum of the Gabor filter responses (in absolute values). If the local maximum is above a threshold, we then declare that there is a sketch within this cell at this orientation, and the sketch is depicted by a bar in the corresponding binary sketch image in Fig. 2. Clearly the sketch image captures a lot of information in the corresponding original image.
Now back to the issue of learning mixture models by EM. We assume that there are Bernoulli templates, and each observed example is a noisy observation of one of the template. The question we want to answer is: given a number of training examples that are noisy observations of the templates, can an EM-type algorithm reliably recover these templates with high probability? The reason we are interested in this question is that it will shed light on unsupervised learning of templates of objects (or their parts) from real images, which is a crucial task for object modeling and recognition in computer vision. Many learning methods are based on fitting mixture models by EM-type algorithms, including the Active Basis model . In the language of the And-Or graph  for object modeling, each template is an And-node, which is a composition of a number of sketches. The mixture of templates is an Or-node, with each template being its child node. So the mixture of the templates is an Or-And structure. The theoretical results in this paper will be useful for us to understand the learning of the Or-And structure from training images.
To answer the above question, we shall generalize the theory of Dasgupta and Shulman 
to Bernoulli distributions, and we shall show that the two-round EM algorithm can learn mixtures of Bernoulli templates with near optimal precision with high probability if the templates are sufficiently different and if the dimensions are sufficiently high.
Generalizing the theory of  from Gaussian mixtures to the mixtures of Bernoulli distributions is far from being straightforward. The sample space is no longer Euclidean, and some results for Gaussian distributions cannot be translated directly into those for the Bernoulli models. So we have to establish a theoretical foundation that is suitable for our purpose. For example, we will need bounds on the tails of the distribution of distances between a template and the mean of binary vectors obtained by perturbing
by Bernoulli noise. Similar bounds for the Gaussian case are easy to obtain because the moment generating function ofis known when is an isotropic Gaussian.
The rest of the paper is organized as follows. Section 2 describes the two-round EM algorithm and states the main theorem. Sections 3 to 4 present theoretical results that lead to the proof of the main theorem. Section 5 illustrates the theoretical results by some experiments on synthetic and real examples. Section 6 concludes with a discussion. In the text, we shall only state the theoretical results. The proofs can be found in the appendix.
2 Two-round EM with performance guarantee
2.1 Model and algorithm
Let be a template. It is an -dimensional binary vector, i.e., . In the example in Fig. 1, . Let be the -th component of , . An example generated by is a noisy version of , and we write . Specifically, let be the -th component of . Then with probability , and with probability , i.e., is the probability of switching a component of , and it defines the level of Bernoulli noise. We assume that . We also assume that the components of are independent given . We call a Bernoulli template because it is binary and is subject to Bernoulli noise.
Let be Bernoulli templates with mixture weights . We assume that is given. Otherwise, can be determined by some model selection criteria such as BIC [4, 7]. Let be noisy observations of these templates, where the noise level is . The probability that is generated by is , and we let . We define to be the expectation of the examples generated by , i.e., where . Let be the set of examples coming from the template .
For two -dimensional vectors and , let be the distance between and . Let be the separation between and , i.e., .
The mixture is called -separated if .
We shall show that if the separation is sufficiently large, then the two-round EM algorithm will reliably recover .
We use the notation to denote the estimated . In the two-round EM, the first round initializes to be randomly selected training examples. The initial number of clusters, , is greater than the true number . Specifically, we let , where is the confidence parameter, i.e., with probability , the algorithm will succeed in recovering the mixture components. According to the coupon collector problem, the examples cover all the clusters with high probability. We estimate the Bernoulli noise level so that based on the statistics of distances between examples derived in Prop. 3. Then we run one more iteration of EM.
After the first iteration, we prune the clusters by a starvation scheme. The pruning process consists of two stages. In the first stage, we remove all the templates whose weights are below a threshold . In the second stage, we keep only templates that are far apart from each other through an inclusion process. Specifically, we start the inclusion process by randomly picking a template. Then in each subsequent step of the inclusion process, we add a template that is farthest away from the selected templates in terms of the minimum distance between the candidate template and the selected templates. We repeat this step until we get templates. We let to index the remaining templates.
After the pruning process, we run another iteration of EM. The estimated templates from this second round EM are already near optimal as we will show.
To be more precise, Algorithm 1 describes the two-round EM. In Step 9 the templates are to be converted to binary by rounding to the nearest integer.
For the convenience of reference, the following summarizes the notation used in this paper:
is the dimension of Bernoulli templates, which generate examples in .
is the number of observations.
is the true number of clusters.
is the level of noise
: the minimum of the mixture weights.
is the -th Bernoulli template
is the set of examples coming from the template .
is the distance between and .
is the separation between the Bernoulli templates,
is the initial number of mixture components . The parameter is the confidence level in Theorem 1.
is the threshold for pruning the clusters learned by the first round.
collects the templates that are initialized from examples in the -th cluster and survive the pruning process after the first round of EM, i.e.
2.3 Main result
Let examples be generated from a mixture of Bernoulli templates under Bernoulli noise of level and mixing weights for all . Let . If the following conditions hold:
The initial number of clusters is
The number of examples is .
The separation is
The dimension is
Then with probability at least , the estimated templates after the round 2 of EM satisfy:
The above theorem states that with high probability, the estimated templates from the two-round EM is nearly as accurate as if we knew the memberships of the examples.
2.4 Sketch of the proof
The proof follows the steps of the two-round EM. We show that after the initialization, with high probability, the initial templates cover all the clusters and the estimated noise level is close to the true noise level . Then after the first round, the estimated templates are likely to be close to the true templates of the same clusters. After the pruning process, we prove that it is very likely that exactly one template is kept for each cluster. Finally after the second round, the estimated templates are proved to be near optimal.
3 Basic facts
We shall first establish some basic facts about the Bernoulli templates perturbed by Bernoulli noise. They are concerned with the distances among templates and their examples.
Let be Bernoulli templates with noise level . We have:
If and then
Let be Bernoulli templates with noise level . We have:
If and then
If and then
then for any
Prop. 2 states that the distance between an example and its template is concentrated around , while the distance between two examples from two different templates is concentrated around . This leads to the following proposition.
Draw samples from a c-separated mixture of Bernoulli templates with mixing weights at least . Let . Then with probability at least
For any we have
For any , , we have
For any we have
Here we employ the notation that means .
Let where are Bernoulli random variables with
are Bernoulli random variables with. Then
(Average of subsets) Draw a set of examples randomly from template with noise level . Then with probability at least for any subset of size at least there is no subset of of size at least whose average has
Prop. 4 states that the sample average is unlikely to deviate too far from .
(Weighted averages) For any finite set of points and weights there exists a subset such that
4 Key steps of the proof
In this section we state the results that hold for the estimated template parameters after each EM iteration. We assume that the following technical conditions hold
These conditions are a subset of the conditions of Theorem 1 that don’t depend on and . They will be referred to in the proofs of the statements of this section.
We also assume that where condition C3 guarantees that . Observe that condition imposes an upper bound on the noise level since . In our experiments this upper bound was between and .
This section analyzes the initial estimates for the parameters before the first round of EM.
With probability at least we have
For each true template , the number of coming from is at least 2.
For each true template , the number of coming from is at most
The noise estimate satisfies
By initializing from more templates than the actual number of clusters, there is a high probability that the estimated templates cover all the clusters.
4.2 First Round of EM
Suppose and , . In the cases when the conclusions of Proposition 3 hold, for any the ratio between the probabilities and is
Prop. 7 states that the first round of EM will likely give higher weights to the templates representing the correct cluster than to a wrong cluster.
In the cases when the conclusions of Proposition 3 hold, any non-starved estimate satisfies with probability
So the estimated template of a cluster is very likely to be close to the true template of this cluster.
We prove that with high probability the pruning step will keep exactly one template from each cluster.
4.4 Second Round of EM
We permute the obtained templates so that .
Suppose that , for all and that conditions hold. Then with probability at least , the estimated templates after the round 2 of EM satisfy:
We are now ready to prove Theorem 1.
Proof of Theorem 1.
From , we get . Also
Take (because of C3). From the dimension condition
we get , so
From the dimension condition we get .
From the condition on the number of examples, we get .
This section illustrates the theoretical results obtained in the previous sections by a simulation study as well as experiments on synthetic image sketches and real images.
5.1 Simulation study
In this section we conduct experiments showing that indeed, the true templates are found with high probability when the conditions of Theorem 1 hold.
We will work with a mixture of two templates, and where the number of 1’s is , to obtain a desired separation in dimension . We experiment with standard EM for 2, 10 and 20 iterations. The standard EM starts from clusters, instead of clusters followed by pruning as in the two-step EM. For the standard EM we also assumed the noise level is a known parameter. All results are obtained from 100 runs.
In the two plots of Fig. 3, the horizontal axis is the minimum weight , and the vertical axis is the separation . The domain for each algorithm is the region above and to the right of the corresponding curve. Two version of the two step EM algorithm were evaluated: the two-step EM, and 10-step version that does 9 EM steps after the pruning step. Five version of the original EM were evaluated, with 2 or 10 iterations, and 1, 5 or 10 random initializations (and selecting from the 5 or 10 obtained results the largest likelihood one as the final result). The first plot is obtained at the noise level , while the second plot is for the noise level . We take the number of observations . For the first plot the dimension is and for the second plot, . One can see that for low noise, the two-step EM works better than the original EM. Also displayed is the domain where the conditions of our theorem are satisfied.
In the four plots from Fig. 4 the horizontal axis is the number of observations and the vertical axis is the dimension . The four plots show the domain where the two-step EM algorithm finds the templates with probability for the levels of noise . The curves corresponding to conditions 2-4 of Theorem 1 and the technical conditions C1-C3 are also displayed, as well as the domain where all conditions of our theorem are satisfied.
From the experiments we observe that the domain where the templates are found with high probability is larger than the domain where the conditions of Theorem 1 hold. The largest discrepancy is in the dimensionality conditions, where the gap between theory and experiments is considerable. This gap could be substantially decreased if tighter bounds could be obtained for Prop 4 and consequently for Prop 8 and Theorem 2.
5.2 Experiments on synthetic image sketches
In this experiment we work with a mixture of two Bernoulli templates, shown in the bottom row of Fig. 5, in a space of dimension . By perturbing the entries with Bernoulli noise of level we obtain images such as those shown in the top row of Fig. 5.
Fig. 6 shows the success rate of finding the two templates exactly using the two-round EM algorithm vs. the number of training examples. The experiments are run for two levels of noise and two mixture weights .
Also shown is the bound from condition 2 of Theorem 1.
The separation between the two templates is quite small , because the two templates share a lot of zero components. So the separation conditions fail in this case. Since we are not in the conditions of the Theorem 1, the bound on the training examples is not expected to hold. We may achieve a better bound if we reduce the dimension while increasing by selecting those features that differentiate the templates. In any case, we see that in the given scenarios the two templates can be recovered with 100% certainty with the two-round EM given sufficiently many examples. So Theorem 1 might hold under milder assumptions than ours.
5.3 Experiments on real images
|Cond. Purity||Cond. Entropy||Log-Likelihood|
|Tow-round EM||2 (1)||0.9402 0.1124||0.1098 0.1862||-110625.6 7914.1||61|
|10 (1)||0.9822 0.0653||0.0351 0.0937||-108460.3 6491.8||76|
|Original EM||2 (1)||0.8511 0.1500||0.2464 0.2193||-115852.1 11079.3||27|
|10 (1)||0.9004 0.1464||0.1555 0.2000||-113476.2 10889.9||43|
|20 (1)||0.8722 0.1572||0.1917 0.2144||-115083.1 10828.8||40|
|100 (1)||0.9051 0.1460||0.1447 0.2006||-113205.1 10809.3||51|
||2 (5)||0.9911 0.0295||0.0260 0.0599||-106268.2 5448.4||77|
|10 (5)||0.9987 0.0053||0.0050 0.0198||-106067.0 5122.5||94|
|20 (5)||0.9956 0.0336||0.0088 0.0493||-106249.3 5540.3||94|
|100 (5)||0.9996 0.0031||0.0017 0.0117||-106045.7 5106.5||98|
||2 (10)||0.9991 0.0054||0.0030 0.0179||-107549.1 5366.8||97|
|10 (10)||1.0000 0.0000||0.0000 0.0000||-107534.8 5377.5||100|
|20 (10)||1.0000 0.0000||0.0000 0.0000||-107534.8 5377.5||100|
|100 (10)||0.9998 0.0022||0.0008 0.0083||-107541.0 5390.3||99|
We also performed experiments on real images. Each image is first convolved with Gabor filters tuned to 16 orientations. Then the image domain is partitioned into equal sized squared cells (the size ranges from pixels to pixels). Within each cell, at each orientation, we pool the maximum of the absolute values of the filter responses. If the maximum is above a threshold, we declare that there is a sketch within this cell at this orientation. Thus each cell produces a binary vector of 16 components. We then concatenate the binary vectors of all the cells into a large binary vector. So each image is transformed into a binary vector.
|Cond. Purity||Cond. Entropy||Log-Likelihood|
|Tow-round EM||2 (1)||0.8918 0.1268||0.2450 0.2230||-398033.9 14285.5||17|
|10 (1)||0.9222 0.1169||0.1705 0.2016||-396428.5 13583.5||32|
|Original EM||2 (1)||0.6853 0.1472||0.6300 0.2429||-408941.0 16116.3||2|
|10 (1)||0.7342 0.1386||0.5151 0.2341||-406125.6 13973.7||2|
|20 (1)||0.7489 0.1484||0.4876 0.2534||-405384.8 15374.4||5|
|100 (1)||0.7493 0.1426||0.4788 0.2497||-405643.4 16290.6||6|
||2 (5)||0.8587 0.1039||0.3459 0.1985||-562506.3 45380.3||6|
|10 (5)||0.9276 0.0881||0.1895 0.1795||-556137.0 45076.8||19|
|20 (5)||0.9027 0.0985||0.2373 0.1861||-558410.6 45627.2||11|
|100 (5)||0.9287 0.0835||0.1834 0.1656||-555971.3 44003.9||20|
||2 (10)||0.9242 0.0731||0.2114 0.1688||-395855.2 11914.2||15|
|10 (10)||0.9618 0.0423||0.1189 0.1021||-394102.3 11968.6||24|
|20 (10)||0.9578 0.0443||0.1344 0.1220||-394252.5 11701.0||25|
|100 (10)||0.9651 0.0459||0.1065 0.1136||-394137.2 12224.4||31|
Evaluation metrics. To evaluate the clustering quality, we use two metrics from : conditional purity and conditional entropy. Given the underlying ground-truth category labels (which are unknown to the algorithm) and the obtained cluster labels , the conditional purity is defined as the mean of the maximum category probabilities for ,
and the conditional entropy is defined as,
where both and are estimated on the training set, and we would expect higher purity and lower entropy for a better clustering algorithm.
We then use the two-round EM algorithm to cluster the images and learn a binary template for each cluster. We compare with the original EM algorithm running for different numbers of iterations ( in the experiments) and starting with desired number of clusters (while the two-round EM starts with clusters and prunes them). A more robust EM could be obtained by starting with many random initialization and choosing the clustering result that has the largest log-likelihood. Such robust versions with different number of initializations are also evaluated in Tables 1, 2 and 3. A ten-round version of the two-round EM (with eight additional EM iterations after the pruning step) is also evaluated. The methods are evaluated in terms of conditional purity and conditional entropy. From the experiments one could see that the two-round EM algorithm can only be outperformed with a five or ten random initializations of the standard EM algorithm. All the results are obtained based on 100 runs.
Fig. 7 and 8 show the results of two experiments (vehicles and animal faces). Table 1 and 2 show the performance comparisons. Table 3 shows the performance all data combined. In the learned templates, the existence of a sketch at each cell is represented by a bar at the center of this cell and at the orientation of the sketch. In each experiment, there are 15 images in each cluster, and the two-round EM is able to separate the clusters perfectly. For the real images, the templates are denser than those in Fig. 5 because the numbers of cells are larger.
|Cond. Purity||Cond. Entropy||Log-Likelihood|
|Tow-round EM||2 (1)||0.8823 0.0982||0.2526 0.1891||-401125.5 28775.3||0|
|10 (1)||0.9030 0.0911||0.1841 0.1506||-398672.2 28364.9||6|
|Original EM||2 (1)||0.7389 0.0995||0.5152 0.1775||-416130.8 32791.7||0|
|10 (1)||0.7744 0.1160||0.4042 0.2001||-412488.2 32696.2||2|
|20 (1)||0.7961 0.1129||0.3669 0.1862||-409443.2 33581.6||2|
|100 (1)||0.7883 0.1265||0.3862 0.2196||-410602.3 34139.1||0|
||2 (5)||0.8468 0.0769||0.3212 0.1364||-402958.6 28555.4||1|
|10 (5)||0.9082 0.0857||0.1793 0.1351||-396123.3 30142.8||9|
|20 (5)||0.9158 0.0790||0.1686 0.1263||-395430.3 27609.8||8|
|100 (5)||0.9022 0.0854||0.1843 0.1290||-396627.6 29355.7||5|
||2 (10)||0.8836 0.0746||0.2669 0.1376||-398549.9 28767.4||1|
|10 (10)||0.9483 0.0602||0.1163 0.0937||-392213.1 27651.5||10|
|20 (10)||0.9504 0.0633||0.1072 0.0985||-392517.4 29030.8||16|
|100 (10)||0.9574 0.0566||0.0992 0.0882||-391457.3 27788.0||10|
Currently we use a very simple sketch detector by thresholding the Gabor filter responses at different orientations. We will design more sophisticated features and associated detectors in future work.
This paper obtains theoretical guarantees on the performance of a two-round EM algorithm for learning mixture of Bernoulli templates, by generalizing the theory of . Unlike the theoretical results for supervised learning, results on unsupervised learning such as clustering are relatively scarce. The results obtained in this paper can be useful for understanding the behavior of EM-type algorithms for unsupervised learning.
In our future work, we shall improve the theoretical results by relaxing the conditions on the separation between the templates as well as the sample size. We shall also generalize Bernoulli templates to more general statistical models for images, such as templates with dependent switching of the binary components, as well as other non-Gaussian models such as exponential family models.
The authors wish to acknowledge support from DARPA MSEE grant FA 8650-11-1-7149 and NSF grant DMS 1007889. The authors thank Maria Pavlovskaia for her help with the experiment on synthetic sketches. Also thanks to Jianwen Xie for assistance with the animal face experiments, and to Prof. Song-Chun Zhu for valuable suggestions.
S. Dasgupta and L.J. Shulman. A Two-round variant of EM for
Proceedings of 16th Conference on Uncertainty in Artificial Intelligence (UAI-2000), 152-159, 2000.
J. G. Daugman. Complete discrete 2-D Gabor transforms by neural networks for image analysis and compressionIEEE Trans. on Acoustics, Speech and Signal Processing, 36, 1169-1179, 1988.
-  A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, B, 39, 1-38, 1977.
-  C. Fraley and A. E. Raftery. Model-based clustering, discriminant analysis, and density estimation. Journal of the American Statistical Association, 97, 611-631, 2002.
-  L.A. Goodman. Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika, 61, 215–231, 1974.
-  X. Huo and D. L. Donoho. Applications of beamlets to detection and extraction of lines, curves and objects in very noisy images. Nonlinear Signal and Image Processing, 2001.
-  G. E. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6, 461-464, 1978.
-  Z Si and H Gong and SC Zhu and YN Wu. Learning active basis models by EM-type algorithms. Statistical Science, 25, 458-475, 2010
V. N. Vapnik.
The nature of Statistical Learning Theory. Springer, 2000.
-  T. Tuytelaars, C. H. Lampert, M. B. Blaschko, and W. Buntine. Unsupervised object discovery: a comparison. International Journal of Computer Vision, 88, 284-302, 2010
-  S. C. Zhu and D. B. Mumford. A stochastic grammar of images. Foundations and Trends in Computer Graphics and Vision, 2, 259–362, 2006.
Proof of Prop. 1.
1. We have
2. Let . Without loss of generality, let where and . Observe that if two random variables are independent then . Then
3. In the case when we have