A provable SVD-based algorithm for learning topics in dominant admixture corpus

by   Trapit Bansal, et al.

Topic models, such as Latent Dirichlet Allocation (LDA), posit that documents are drawn from admixtures of distributions over words, known as topics. The inference problem of recovering topics from admixtures, is NP-hard. Assuming separability, a strong assumption, [4] gave the first provable algorithm for inference. For LDA model, [6] gave a provable algorithm using tensor-methods. But [4,6] do not learn topic vectors with bounded l_1 error (a natural measure for probability vectors). Our aim is to develop a model which makes intuitive and empirically supported assumptions and to design an algorithm with natural, simple components such as SVD, which provably solves the inference problem for the model with bounded l_1 error. A topic in LDA and other models is essentially characterized by a group of co-occurring words. Motivated by this, we introduce topic specific Catchwords, group of words which occur with strictly greater frequency in a topic than any other topic individually and are required to have high frequency together rather than individually. A major contribution of the paper is to show that under this more realistic assumption, which is empirically verified on real corpora, a singular value decomposition (SVD) based algorithm with a crucial pre-processing step of thresholding, can provably recover the topics from a collection of documents drawn from Dominant admixtures. Dominant admixtures are convex combination of distributions in which one distribution has a significantly higher contribution than others. Apart from the simplicity of the algorithm, the sample complexity has near optimal dependence on w_0, the lowest probability that a topic is dominant, and is better than [4]. Empirical evidence shows that on several real world corpora, both Catchwords and Dominant admixture assumptions hold and the proposed algorithm substantially outperforms the state of the art [5].


page 1

page 2

page 3

page 4


A Spectral Algorithm for Latent Dirichlet Allocation

The problem of topic modeling can be seen as a generalization of the clu...

Source-LDA: Enhancing probabilistic topic models using prior knowledge sources

A popular approach to topic modeling involves extracting co-occurring n-...

Generalized Topic Modeling

Recently there has been significant activity in developing algorithms wi...

Spectral Learning for Supervised Topic Models

Supervised topic models simultaneously model the latent topic structure ...

VSEC-LDA: Boosting Topic Modeling with Embedded Vocabulary Selection

Topic modeling has found wide application in many problems where latent ...

Provable Algorithms for Inference in Topic Models

Recently, there has been considerable progress on designing algorithms w...

Necessary and Sufficient Conditions and a Provably Efficient Algorithm for Separable Topic Discovery

We develop necessary and sufficient conditions and a novel provably cons...

1 Introduction

Topic models [1] assume that each document in a text corpus is generated from an ad-mixture of topics, where, each topic is a distribution over words in a Vocabulary. An admixture is a convex combination of distributions. Words in the document are then picked in i.i.d. trials, each trial has a multinomial distribution over words given by the weighted combination of topic distributions. The problem of inference, recovering the topic distributions from such a collection of documents, is provably NP-hard. Existing literature pursues techniques such as variational methods [2] or MCMC procedures [3]

for approximating the maximum likelihood estimates.

Given the intractability of the problem one needs further assumptions on topics to derive polynomial time algorithms which can provably recover topics. A possible (strong) assumption is that each document has only one topic but the collection can have many topics. A document with only one topic is sometimes referred as a pure topic document. [7] proved that a natural algorithm, based on SVD, recovers topics when each document is pure and in addition, for each topic, there is a set of words, called primary words, whose total frequency in that topic is close to 1. More recently, [6] show using tensor methods that if the topic weights have Dirichlet distribution, we can learn the topic matrix. Note that while this allows non-pure documents, the Dirichlet distribution gives essentially uncorrelated topic weights.

In an interesting recent development [4, 5] gave the first provable algorithm which can recover topics from a corpus of documents drawn from admixtures, assuming separability. Topics are said to be separable if in every topic there exists at least one Anchor word. A word in a topic is said to be an an Anchor word for that topic if it has a high probability in that topic and zero probability in remaining topics. The requirement of high probability in a topic for a single word is unrealistic.

Our Contributions:

Topic distributions, such as those learnt in LDA, try to model the co-occurrence of a group of words which describes a theme. Keeping this in mind we introduce the notion of Catchwords. A group of words are called Catchwords of a topic, if each word occurs strictly more frequently in the topic than other topics and together they have high frequency. This is a much weaker assumption than separability. Furthermore we observe, empirically, that posterior topic weights assigned by LDA to a document often have the property that one of the weights is significantly higher than the rest. Motivated by this observation, which has not been exploited by topic modeling literature, we suggest a new assumption. It is natural to assume that in a text corpus, a document, even if it has multiple themes, will have an overarching dominant theme. In this paper we focus on document collections drawn from dominant admixtures. A document collection is said to be drawn from a dominant admixture if for every document, there is one topic whose weight is significantly higher than other topics and in addition, for every topic, there is a small fraction of documents which are nearly purely on that topic. The main contribution of the paper is to show that under these assumptions, our algorithm, which we call TSVD , indeed provably finds a good approximation in total error to the topic matrix. We prove a bound on the error of our approximation which does not grow with dictionary size , unlike [5] where the error grows linearly with .

Empirical evidence shows that on semi-synthetic corpora constructed from several real world datasets, as suggested by [5], TSVD substantially outperforms the state of the art [5]. In particular it is seen that compared to [5] TSVD gives % lower error in terms of recovery on % of the topics.

Problem Definition:

will denote respectively, the number of words in the dictionary, number of topics and number of documents. are large, whereas, is to be thought of as much smaller. Let . For each topic, there is a fixed vector in giving the probability of each word in that topic. Let be the matrix with these vectors as its columns.

Documents are picked in i.i.d. trials. To pick document , one first picks a vector of topic weights according to a fixed distribution on . Let be the weighted combination of the topic vectors. Then the words of the document are picked in i.i.d. trials; each trial picks a word according to the multinomial distribution with as the probabilities. All that is given as data is the frequency of words in each document, namely, we are given the matrix , where  Note that , where, the expectation is taken entry-wise.

In this paper we consider the problem of finding given .

2 Previous Results

In this section we review literature related to designing provable algorithms for topic models. For an overview of topic models we refer the reader to the excellent survey[1]. Provable algorithms for recovering topic models was started by [7]. Latent Semantic Indexing(LSI) [8] remains a successful method for retrieving similar documents by using SVD. [7] showed that one can recover from a collection of documents, with pure topics, by using SVD based procedure under the additional Primary Words Assumption. [6] showed that in the admixture case, if one assumes Dirichlet distribution for the topic weights, then, indeed, using tensor methods, one can learn to error provided some added assumptions on numerical parameters like condition number are satisfied.

The first provably polynomial time algorithm for admixture corpus was given in [4, 5]. For a topic , a word is an anchor word if

Theorem 2.1

[4] If every topic has an anchor word, there is a polynomial time algorithm that returns an such that with high probability,

where, is the condition number of , is the minimum expected weight of a topic and is the number of words in each document.

Note that the error grows linearly in the dictionary size , which is often large. Note also the dependence of on parameters , which is, and on , which is . If, say, the word “run” is an anchor word for the topic “baseball” and , then the requirement is that every 10 th word in a document on this topic is “run”. This seems too strong to be realistic. It would be more realistic to ask that a set of words like - “run”, “hit”, “score”, etc. together have frequency at least 0.1 which is what our catchwords assumption does.

3 Learning Topics from Dominant Admixtures

Informally, a document is said to be drawn from a Dominant Admixture if the document has one dominant topic. Besides its simplicity, we show empirical evidence from real corpora to demonstrate that topic dominance is a reasonable assumption. The Dominant Topic assumption is weaker than the Pure Topic assumption. More importantly SVD based procedures proposed by [7] will not apply. Inspired by the Primary words assumption we introduce the assumption that each topic has a set of Catchwords which individually occur more frequently in that topic than others. This is again a much weaker assumption than both Primary Words and Anchor Words assumptions and can be verified experimentally. In this section we establish that by applying SVD on a matrix, obtained by thresholding the word-document matrix, and subsequent means clustering can learn topics having Catchwords from a Dominant Admixture corpus.

3.1 Assumptions: Catchwords and Dominant admixtures

Let be non-negative reals satisfying:


Dominant topic Assumption (a) For , document has a dominant topic such that

(b)For each topic , there are at least documents in each of which topic has weight at least .

Catchwords Assumption: There are disjoint sets of words - such that with defined in (9)


Part (b.) of the Dominant Topic Assumption is in a sense necessary for “identifiability” - namely for the model to have a set of document vectors so that every document vector is in the convex hull of these vectors. The Catchwords assumption is natural to describe a theme as it tries to model a unique group of words which is likely to co-occur when a theme is expressed. This assumption is close to topics discovered by LDA like models, which try to model of co-occurence of words. If , then, the assumption (5) says . In fact if , we do not expect to see word (in topic ), so it cannot be called a catchword at all.

A slightly different (but equivalent) description of the model will be useful to keep in mind. What is fixed (not stochastic) are the matrices and the distribution of the weight matrix . To pick document , we can first pick the dominant topic in document and condition the distribution of on this being the dominant topic. One could instead also think of being picked from a mixture of distributions. Then, we let and pick the words of the document in i.i.d multinomial trials as before. We will assume that

where, is the probability of topic being dominant. This is only approximately valid, but the error is small enough that we can disregard it.

For , let be the probability that and and the corresponding “empirical probability”:


Note that is a real number, whereas,

is a random variable with

. We need a technical assumption on the (which is weaker than unimodality).

No-Local-Min Assumption: We assume that does not have a local minimum, in the sense:


The justification for the this assumption is two-fold. First, generally, Zipf’s law kind of behaviour where the number of words plotted against relative frequencies declines as a power function has often been observed. Such a plot is monotonically decreasing and indeed satisfies our assumption. But for Catchwords, we do not expect this behaviour - indeed, we expect the curve to go up initially as the relative frequency increases, then reach a maximum and then decline. This is a unimodal function and also satisfies our assumption. Indeed, we have empirically observed, see EXPTS, that these are essentially the only two behaviours.

Relative sizes of parameters Before we close the section we discuss the values of the parameters are in order. Here,

is large. For asymptotic analysis, we can think of it as going to infinity.

is also large and can be thought of as going to infinity. [In fact, if , then, intuitively, we see that there is no use of a corpus of more than constant size - since our model has i.i.d. documents, intuitively, the number of samples we need should depend mainly on ]. is (much) smaller, but need not be constant.

refers to a generic constant independent of ; its value may be different in different contexts.

3.2 The TSVD Algorithm

Existing SVD based procedures for clustering on raw word-document matrices fail because the spread of frequencies of a word within a topic is often more (at least not significantly less) than the gap between the word’s frequencies in two different topics. Hypothetically the frequency for the word run, in the topic Sports, may range from say 0.01 on up. But in other topics, it may range from 0 to 0.005 say. The success of the algorithm will lie on correctly identifying the dominant topics such as sports by identifying that the word run has occurred with high frequency. In this example, the gap (0.01-0.005) between Sports and other topics is less than the spread within Sports (1.0-0.01), so a 2-clustering approach (based on SVD) will split the topic Sports into two. While this is a toy example, note that if we threshold the frequencies at say 0.01, ideally, sports will be all above and the rest all below the threshold, making the succeeding job of clustering easy.

There are several issues in extending beyond the toy case. Data is not one-dimensional. We will use different thresholds for each word; word will have a threshold . Also, we have to compute . Ideally we would not like to split any , namely, we would like that for each and and each , either most have or most have . We will show that our threshold procedure indeed achieves this. One other nuance: to avoid conditioning, we split the data into two parts and , compute the thresholds using and actually do the thresholding on . We will assume that the intial had columns, so each part now has columns. Also, partitions the columns of as well as those of . The columns of thresholded matrix are then clustered, by a technique we call Project and Cluster, namely, we project the columns of to its dimensional SVD subspace and cluster in the projection. The projection before clustering has recently been proven [9] (see also [10]) to yield good starting cluster centers. The clustering so found is not yet satisfactory. We use the classic Lloyd’s -means algorithm proposed by [12]. As we will show, the partition produced after clustering, of is close to the partition induced by the Dominant Topics, . Catchwords of topic are now (approximately) identified as the most frequently occurring words in documents in . Finally, we identify nearly pure documents in (approximately) as the documents in which the catchwords occur the most. Then we get an approximation to by averaging these nearly pure documents. We now describe the precise algorithm.

3.3 Topic recovery using Thresholded SVD

Threshold SVD based K-means (TSVD)

  1. Randomly partition the columns of into two matrices and of  columns each.

  2. Thresholding

    1. Compute Thresholds on For each , let be the highest value of such that

    2. Do the thresholding on

  3. SVD Find the best rank approximation to .

  4. Identify Dominant Topics

    1. Project and Cluster Find (approximately) optimal means clustering of the columns of .

    2. Lloyd’s Algorithm Using the clustering found in Step 4(a) as the starting clustering, apply Lloyd’s algorithm means algorithm to the columns of (, not ).

    3. Let be the partition of corresponding to the clustering after Lloyd’s. //*Will prove that *//

  5. Identify Catchwords

    1. For each , compute the th highest element of .

    2. Let where, .

  6. Find Topic Vectors Find the highest among all and return the average of these as our approximation to .

Theorem 3.1

Main Theorem Under the Dominant Topic, Catchwords and No-Local-Min assumptions, the algorithm succeeds with high probability in finding an so that

A note on the sample complexity is in order. Notably, the dependence of on is best possible (namely ) within logarithmic factors, since, if we had fewer than documents, a topic which is dominant with probability only may have none of the documents in the collection. The dependence of on needs to be at least : to see this, note that we only assume that there are

nearly pure documents on each topic. Assuming we can find this set (the algorithm approximately does), their average has standard deviation of about

in coordinate . If topic vector has entries, each of size , to get an approximation of to error , we need the per coordinate error to be at most which implies . Note that to get comparable error in [4], we need a quadratic dependence on .

There is a long sequence of Lemmas to prove the theorem. The Lemmas and the proofs are given in Appendix. The essence of the proof lies in proving that the clustering step correctly identifies the partition induced by the dominant topics. For this, we take advantage of a recent development on the means algorithm from [9] [see also [10]], where, it is shown that under a condition called the Proximity Condition, Lloyd’s

means algorithm starting with the centers provided by the SVD-based algorithm, correctly identifies almost all the documents’ dominant topics. We prove that indeed the Proximity Condition holds. This calls for machinery from Random Matrix theory (in particular bounds on singular values). We prove that the singular values of the thresholded word-document matrix are nicely bounded. Once the dominant topic of each document is identified, we are able to find the Catchwords for each topic. Now, we rely upon part (b.) of the Dominant Topic assumption : that is there is a small fraction of nearly Pure Topic-documents for each topic. The Catchwords help isolate the nearly pure-topic documents and hence find the topic vectors. The proofs are complicated by the fact that each step of the algorithm induces conditioning on the data- for example, after clustering, the document vectors in one cluster are not anymore independent.

4 Experimental Results

We compare the thresholded SVD based k-means (TSVD222Resources available at: http://mllab.csa.iisc.ernet.in/tsvd) algorithm 3.2 with the algorithms of [5], Recover-KL and Recover-L2, using the code made available by the authors333http://www.cs.nyu.edu/~halpern/files/anchor-word-recovery.zip. We first provide empirical support for the algorithm assumptions in Section 3.1, namely the dominant topic and the catchwords assumption. Then we show on 4 different semi-synthetic data that TSVD provides as good or better recovery of topics than the Recover algorithms. Finally on real-life datasets, we show that the algorithm performs as well as [5] in terms of perplexity and topic coherence.

Implementation Details:

TSVD parameters () are not known in advance for real corpus. We tested empirically for multiple settings and the following values gave the best performance. Thresholding parameters used were: , . For finding the catchwords, in step 5. For finding the topic vectors (step 6), taking the top 50% () gave empirically better results. The same values were used on all the datasets tested. The new algorithm is sensitive to the initialization of the first k-means step in the projected SVD space. To remedy this, we run 10 independent random initializations of the algorithm with K-Means++ [13] and report the best result.

Datasets: We use four real word datasets in the experiments. As pre-processing steps we removed standard stop-words, selected the vocabulary size by term-frequency and removed documents with less than 20 words. Datasets used are: (1) NIPS444http://archive.ics.uci.edu/ml/datasets/Bag+of+Words: Consists of 1,500 NIPS full papers, vocabulary of 2,000 words and mean document length 1023. (2) NYT44footnotemark: 4: Consists of a random subset of 30,000 documents from the New York Times dataset, vocabulary of 5,000 words and mean document length 238. (3) Pubmed44footnotemark: 4: Consists of a random subset of 30,000 documents from the Pubmed abstracts dataset, vocabulary of 5,030 words and mean document length 58. (4) 20NewsGroup555http://qwone.com/~jason/20Newsgroups (20NG): Consist of 13,389 documents, vocabulary of 7,118 words and mean document length 160.

4.1 Algorithm Assumptions

To check the dominant topic and catchwords assumptions, we first run 1000 iterations of Gibbs sampling on the real corpus and learn the posterior document-topic distribution () for each document in the corpus (by averaging over 10 saved-states separated by 50 iterations after the 500 burn-in iterations). We will use this posterior document-topic distribution as the document generating distribution to check the two assumptions.

Dominant topic assumption: Table 1 shows the fraction of the documents in each corpus which satisfy this assumption with (minimum probability of dominant topic) and (maximum probability of non-dominant topics). The fraction of documents which have almost pure topics with highest topic weight at least 0.95 () is also shown. The results indicate that the dominant topic assumption is well justified (on average 64% documents satisfy the assumption) and there is also a substantial fraction of documents satisfying almost pure topic assumption.

Catchwords assumption: We first find a -clustering of the documents

by assigning all documents which have highest posterior probability for the same topic into one cluster. Then we use step 5 of TSVD (Algorithm

3.2) to find the set of catchwords for each topic-cluster, i.e. , with the parameters: , (taking into account constraints in Section 3.1, ). Table 1 reports the fraction of topics with non-empty set of catchwords and the average per topic frequency of the catchwords666. Results indicate that most topics on real data contain catchwords (Table 1, second-last column). Moreover, the average per-topic frequency of the group of catchwords for that topic is also quite high (Table 1, last column).

No-Local-Min Assumption: To provide support and intuition for the local-min assumption we consider the quantity , in (7). Recall that , we will analyze the behavior of as a function of for some topics and words . As defined, we need a fixed to check this assumption and so we generate semi-synthetic data with a fixed from LDA model trained on the real NYT corpus (as explained in Section 4.2.1). We find catchwords and the sets as in the catchwords assumption above and plot separately for some random catchwords and non-catchwords by fixing some random . Figure 1 shows the plots. As explained in 3.1, the plots are monotonically decreasing for non-catchwords and satisfy the assumption. On the other hand, the plots for catchwords are almost unimodal and also satisfy the assumption.

Corpus % s with Dominant % s with Pure % Topics CW Mean
Topics () Topics () with CW Frequency
NIPS 1500 50 56.6% 2.3% 96% 0.05
NYT 30000 50 63.7% 8.5% 98% 0.07
Pubmed 30000 50 62.2% 5.1% 78% 0.05
20NG 13389 20 74.1% 39.5% 85% 0.06
Table 1: Algorithm Assumptions. For dominant topic assumption, fraction of documents with satisfy the assumption for are shown. % documents with almost pure topics (, i.e. pure) are also shown. Last two columns show results for catchwords (CW) assumption.
Figure 1: Plot of for some random catchwords (left) and non-catchwords (right). Each of three plots for catchword is for one topic () with two random catchwords () for each topic and each plot on right is for one non-catchword () with curves for multiple topics ().

4.2 Empirical Results

4.2.1 Topic Recovery on Semi-Synthetic Data

Semi-synthetic Data: Following [5], we generate semi-synthetic corpora from LDA model trained by MCMC, to ensure that the synthetic corpora retain the characteristics of real data. Gibbs sampling is run for 1000 iterations on all the four datasets and the final word-topic distribution is used to generate varying number of synthetic documents with document-topic distribution drawn from a symmetric Dirichlet with hyper-parameter 0.01. For NIPS, NYT and Pubmed we use topics, for 20NewsGroup , and mean document lengths of 1000, 300, 100 and 200 respectively. Note that the synthetic data is not guaranteed to satisfy dominant topic assumption for every document (on average about 80% documents satisfy the assumption for value of tested in Section 4.1)

Topic Recovery: We learn the word-topic distributions () for the semi-synthetic corpora using TSVD and the Recover algorithms of [5]. Given these learned topic distributions and the original data-generating distributions (), we align the topics of and by bipartite matching and rearrange the columns of in accordance to the matching with . Topic recovery is measured by the average of the error across topics (called reconstruction error [5]), , defined as: .

We report reconstruction error in Table 2 for TSVD and the Recover algorithms, Recover-L2 and Recover-KL. TSVD has smaller error on most datasets than the Recover-KL algorithm. We observed performance of TSVD to be always better than Recover-L2. Best performance is observed on NIPS which has largest mean document length, indicating that larger leads to better recovery. Results on 20NG are slightly worse than Recover-KL for small sample size (though better than Recover-L2), but the difference is small. While the values in Table 2 are averaged values, Figure 2 shows that TSVD algorithm achieves much better topic recovery (27% improvement in error over Recover-KL) for majority of the topics (90%) on most datasets.

Corpus Documents Recover-L2 Recover-KL TSVD % Improvement
NIPS 40,000 0.342 0.308 0.115 62.7%
50,000 0.346 0.308 0.145 52.9%
60,000 0.346 0.311 0.131 57.9%
Pubmed 40,000 0.388 0.332 0.288 13.3%
50,000 0.378 0.326 0.280 14.1%
60,000 0.372 0.328 0.284 13.4%
20NG 40,000 0.126 0.120 0.124 -3.3%
50,000 0.118 0.114 0.113 0.9%
60,000 0.114 0.110 0.106 3.6%
NYT 40,000 0.214 0.208 0.195 6.3%
50,000 0.211 0.206 0.185 10.2%
60,000 0.205 0.200 0.194 3.0%
Table 2: L1 reconstruction error on various semi-synthetic datasets. Last column is percent improvement over Recover-KL (best performing Recover algorithm).
Figure 2: Histogram of error across topics for 40,000 synthetic documents. TSVD (blue, solid border) gets better recovery on most topics (

) for most datasets (leaving small number of outliers) than

Recover-KL (green, dashed border).

4.2.2 Topic Recovery on Real Data


A standard quantitative measure used to compare topic models and inference algorithms is perplexity [2]. Perplexity of a set of test documents, where each document consists of words, denoted by , is defined as: . To evaluate perplexity on real data, the held-out sets consist of 350 documents for NIPS, 10000 documents for NYT and Pubmed, and 6780 documents for 20NewsGroup. Table 3 shows the results of perplexity on the 4 datasets. TSVD gives comparable perplexity with Recover-KL, results being slightly better on NYT and 20NewsGroup which are larger datasets with moderately high mean document lengths.

Topic Coherence:

[11] proposed Topic Coherence as a measure of semantic quality of the learned topics by approximating user experience of topic quality on top words of a topic. Topic coherence is defined as , where is the document frequency of a word , is the document frequency of and together, and is a small constant. We evaluate TC for the top 5 words of the recovered topic distributions and report the average and standard deviation across topics. TSVD gives comparable results on Topic Coherence (see Table 3).

Topics on Real Data:

Table 4 shows the top 5 words of all 50 matched pair of topics on NYT dataset for TSVD, Recover-KL and Gibbs sampling. Most of the topics recovered by TSVD are more closer to Gibbs sampling topics. Indeed, the total average error with topics from Gibbs sampling for topics from TSVD is 0.034, whereas for Recover-KL it is 0.047 (on the NYT dataset).

Corpus Perplexity Topic Coherence
NIPS 754 749 835 -86.4 24.5 -88.6 22.7 -65.2 29.4
NYT 1579 1685 1555 -105.2 25.0 -102.1 28.2 -107.6 25.7
Pubmed 1188 1203 1307 -94.0 22.5 -94.4 22.5 -84.5 28.7
20NG 2431 2565 2390 -93.7 13.6 -89.4 20.7 -90.4 27.0
Table 3: Perplexity and Topic Coherence. R-KL is Recover-KL, R-L2 is Recover-L2. Standard deviation for topic coherence across topics is also shown.

Summary: We evaluated the proposed algorithm, TSVD, rigorously on multiple datasets with respect to the state of the art (Recover), following the evaluation methodology of [5]. In Table 2 we show that the L1 reconstruction error for the new algorithm is small and on average 19.6% better than the best results of the Recover algorithms [5]. We also demonstrate that on real datasets the algorithm achieves comparable perplexity and topic coherence to Recover (Table 3. Moreover, we show on multiple real datasets that the algorithm assumptions are well justified in practice.


Real world corpora often exhibits the property that in every document there is one topic dominantly present. A standard SVD based procedure will not be able to detect these topics, however TSVD, a thresholded SVD based procedure, as suggested in this paper, discovers these topics. While SVD is time-consuming, there have been a host of recent sampling-based approaches which make SVD easier to apply to massive corpora which may be distributed among many servers. We believe that apart from topic recovery, thresholded SVD can be applied even more broadly to similar problems, such as matrix factorization, and will be the basis for future research.

TSVD Recover-KL Gibbs
zzz_elian zzz_miami boy father zzz_cuba zzz_elian boy zzz_miami father family zzz_elian zzz_miami boy father zzz_cuba
cup minutes add tablespoon oil cup minutes tablespoon add oil cup minutes add tablespoon oil
game team yard zzz_ram season game team season play zzz_ram team season game coach zzz_nfl
book find british sales retailer book find school woman women book find woman british school
run inning hit season game run season game inning hit run season game hit inning
church zzz_god religious jewish christian pope church book jewish religious religious church jewish jew zzz_god
patient drug doctor cancer medical patient drug doctor percent found patient doctor drug medical cancer
music song album musical band black reporter zzz_new_york zzz_black show music song album band musical
computer software system zzz_microsoft company web www site cookie cookies computer system software technology mail
house dog water hair look room show look home house room look water house hand
zzz_china trade zzz_united_states nuclear official zzz_china zzz_taiwan government trade zzz_party zzz_china zzz_united_states zzz_u_s zzz_clinton zzz_american
zzz_russian war rebel troop military zzz_russian zzz_russia war zzz_vladimir_putin rebel war military zzz_russian soldier troop
officer police case lawyer trial zzz_ray_lewis police case officer death police officer official case investigation
car driver wheel race vehicles car driver truck system model car driver truck vehicle wheel
show network zzz_abc zzz_nbc viewer con zzz_mexico son federal mayor show television network series zzz_abc
com question information zzz_eastern sport com information question zzz_eastern sport com information daily question zzz_eastern
book author writer com reader zzz_john_rocker player team right braves book word writer author wrote
zzz_al_gore zzz_bill_bradley campaign president democratic zzz_al_gore zzz_bill_bradley campaign president percent zzz_al_gore campaign zzz_bill_bradley president democratic
actor film play movie character goal play team season game film movie award actor zzz_oscar
school student teacher district program school student program million children school student teacher program children
tax taxes cut billion plan zzz_governor_bush tax campaign taxes plan tax plan billion million cut
percent stock market fund investor million percent tax bond fund stock market percent fund investor
team player season coach zzz_nfl team season player coach zzz_cowboy team player season coach league
family home friend room school look gun game point shot family home father son friend
primary zzz_mccain voter zzz_john_mccain zzz_bush zzz_john_mccain zzz_george_bush campaign republican voter zzz_john_mccain zzz_george_bush campaign zzz_bush zzz_mccain
zzz_microsoft court company case law zzz_microsoft company computer system software zzz_microsoft company window antitrust government
company million percent shares billion million company stock percent shares company million companies business market
site web sites com www web site zzz_internet company com web site zzz_internet online sites
scientist human cell study researcher dog quick jump altered food plant human food product scientist
baby mom percent home family mate women bird film idea women look com need telegram
point game half shot team point game team season zzz_laker game point team play season
zzz_russia zzz_vladimir_putin zzz_russian zzz_boris_yeltsin zzz_moscow zzz_clinton government zzz_pakistan zzz_india zzz_united_states government political election zzz_vladimir_putin zzz_russia
com zzz_canada www fax information chocolate food wine flavor buy www com hotel room tour
room restaurant building fish painting zzz_kosovo police zzz_serb war official building town area resident million
loved family show friend play film show movie music book film movie character play director
prices percent worker oil price percent stock market economy prices percent prices economy market oil
million test shares air president air wind snow shower weather water snow weather air scientist
zzz_clinton flag official federal zzz_white_house zzz_bradley zzz_al_gore campaign zzz_gore zzz_clinton zzz_clinton president gay mayor zzz_rudolph_giuliani
files article computer art ball show film country right women art artist painting museum show
con percent zzz_mexico federal official official zzz_iraq government zzz_united_states oil zzz_mexico drug government zzz_united_states mexican
involving book film case right test women study student found plane flight passenger pilot zzz_boeing
zzz_internet companies company business customer company companies deal zzz_internet zzz_time_warner media zzz_time_warner television newspaper cable
zzz_internet companies company business customer newspaper zzz_chronicle zzz_examiner zzz_hearst million million money worker company pay
goal play games king game zzz_tiger_wood shot tournament tour player zzz_tiger_wood tour tournament shot player
zzz_american zzz_united_states zzz_nato camp war zzz_israel zzz_lebanon peace zzz_syria israeli zzz_israel peace palestinian talk israeli
team season game player play team game point season player race won win fight team
reporter zzz_earl_caldwell zzz_black black look corp group list oil meeting black white zzz_black hispanic reporter
campaign zzz_republican republican zzz_party primary zzz_bush zzz_mccain campaign republican voter gun bill law zzz_congress legislation
zzz_bush zzz_mccain campaign primary republican flag black zzz_confederate right group flag zzz_confederate zzz_south_carolina black zzz_south
zzz_john_mccain campaign zzz_george_bush zzz_bush republican official government case officer security court law case lawyer right
Table 4: Top 5 words of matched topic pairs for TSVD, Recover-KL and Gibbs sampling. Catchwords and anchor words in top 5 words are highlighted for TSVD and Recover-KL


  • [1] Blei, D. Introduction to probabilistic topic models. Communications of the ACM, pp. 77–84, 2012.
  • [2] Blei, D., Ng, A., and Jordan, M. Latent Dirichlet allocation.

    Journal of Machine Learning Research

    , pp. 3:993–1022, 2003.
    Preliminary version in Neural Information Processing Systems 2001.
  • [3] Griffiths, T. L. and Steyvers, M. Finding scientific topics. Proceedings of the National Academy of Sciences, 101:5228–5235, 2004.
  • [4] Arora, S., Ge, R., and Moitra, A. Learning topic models – going beyond SVD. In Foundations of Computer Science, 2012b.
  • [5] Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag, D., Wu, Y., and Zhu M. A practical algorithm for topic modeling with provable guarantees. In Internation Conference on Machine Learning, 2013
  • [6] Anandkumar, A., Foster, D., Hsu, D., Kakade, S., and Liu, Y. A Spectral Algorithm for Latent Dirichlet Allocation In Neural Information Processing Systems, 2012.
  • [7] Papadimitriou, C., Raghavan, P., Tamaki H., and Vempala S. Latent semantic indexing: a probabilistic analysis. Journal of Computer and System Sciences, pp. 217–235, 2000. Preliminary version in PODS 1998.
  • [8] Deerwester, S., Dumais, S., Landauer, T., Furnas, G., and Harshman, R. Indexing by latent semantic analysis. Journal of the American Society for Information Science, pp. 391–407, 1990.
  • [9] Kumar, A., and Kannan, R. Clustering with spectral norm and the k-means algorithm. In Foundations of Computer Science, 2010
  • [10] Awashti, P., and Sheffet, O. Improved spectral-norm bounds for clustering. In Proceedings of Approx/Random, 2012.
  • [11] Mimno, D., Wallach, H., Talley, E., Leenders, M. and McCallum, A. Optimizing semantic coherence in topic models. In

    Empirical Methods in Natural Language Processing

    , pp. 262–272, 2011.
  • [12] Lloyd, Stuart P. Least squares quantization in PCM, IEEE Transactions on Information Theory 28 (2): 129–137,1982.
  • [13] Arthur, D., and Vassilvitskii, S. K-means++: The advantages of careful seeding. In Proceedings of ACM-SIAM symposium on Discrete algorithms, pp. 1027–1035, 2007
  • [14] McDiarmid, C. On the method of Bounded Differences. Surveys in Combinatorics: London Math. Soc. Lecture Note Series 141. Cambridge University Press., 1989.
  • [15] Vershynin R. Introduction to non-asymptotic analysis of random matrices. In ArXiv:1011.3027v6 [math.PR] 4 Oct 2011

Appendix A Line of Proof

We describe the Lemmas we prove to establish the result. The detailed proofs are in the Section B.

a.1 General Facts

We start with a consequence of the no-local-minimum assumption. We use that assumption solely through this Lemma.

Lemma A.1

Let be as in (6). If for some and , and also then, .

Next, we state a technical Lemma which is used repeatedly. It states that for every , the empirical probability that is close to the true probability. Unsurprisingly, we prove it using H-C. But we will state a consequence in the form we need in the sequel.

Lemma A.2

Let and be as in (6) and (7). We have

From this it follows that with probability at least ,

a.1.1 Properties of Thresholding

Say that a threshold “splits” if has a significant number of with and also a significant number of with . Intuitively, it would be desirable if no threshold splits any , so that, in , for each , either most have or most have . We now prove that this is indeed the case with proper bounds. We henceforth refer to the conclusion of the Lemma below by the mnemonic “no threshold splits any ”.

Lemma A.3

(No Threshold Splits any ) For a fixed , with probability at least , the following holds:

Let be a matrix whose columns are given by

’s columns corresponding to all are the same. The entries of the matrix are fixed (real numbers) once we have (and the thresholds are determined). Note: We have “integrated out ”, i.e.,

(So, think of for ’s columns being picked first from which is calculated. for columns of are not yet picked until the are determined.) But are random variables before we fix . The following Lemma is a direct consequence of “no threshold splits any ”.

Lemma A.4

Let . With probability at least (over the choice of ):


where, .

So far, we have proved that for every , the threshold does not split any . But this is not sufficient in itself to be able to cluster (and hence identify the ), since, for example, this alone does not rule out the extreme cases that for most in every , is above the threshold (whence for almost all ) or for most in no is above the threshold, whence, for almost all . Both these extreme cases would make us loose all the information about due to thresholding; this scenario and milder versions of it have to be proven not to occur. We do this by considering how thresholds handle catchwords. Indeed we will show that for a catchword , a has above the threshold and a has below the threshold. Both statements will only hold with high probability, of course and using this, we prove that and are not too close for in different ’s. For this, we need the following Lemmas.

Lemma A.5

For , and , we have with ,

Lemma A.6

With probability at least , we have

a.1.2 Proximity

Next, we wish to show that clustering as in TSVD identifies the dominant topics correctly for most documents, i.e., that for all . For this, we will use a theorem from [9] [see also [10]] which in this context says:

Theorem A.7

If all but a fraction of the the satisfy the “proximity condition”, then TSVD identifies the dominant topic in all but fraction of the documents correctly after polynomial number of iterations.

To describe the proximity condition, first let be the maximum over all directions of the square root of the mean-squared distance of to , i.e.,

The parameter should remind the reader of standard deviation, which is indeed what this is, since . Our random variables being dimensional vectors, we take the maximum standard deviation in any direction.

  • is said to satisfy the proximity condition with respect to , if for each and each and and each and , the projection of onto the line joining and is closer to by at least

    than it is to . [Here, is a constant.]

To prove proximity, we need to bound . This will be the task of the subsection B.1 which relies heavily on Random Matrix Theory.

Appendix B Proofs of Correctness

We start by recalling the Höffding-Chernoff (H-C) inequality in the form we use it.

Lemma B.1

Höffding-Chernoff If is the average of independent random variables with values in and , then, for an ,

  • (of Lemma A.1) Abbreviate by . We claim that either (i) or (ii) To see this, note that if both (i) and (ii) fail, we have and with . But then there has to be a local minimum of between and . If (i) holds, clearly, and so the lemma follows. So, also if (ii) holds.

  • (of Lemma A.2) Note that where, is the indicator variable of . and we apply H-C with and . We have , as is easily seen by calculating the roots of the quadratic . Thus we get the claimed for . Note that the same proof applies for as well as .

    To prove the second assertion, let and , then, satisfies the quadratic inequalities:

    By bounding the roots of these quadratics, it is easy to see the second assertion after some calculation.

  • (of Lemma A.3) Note that is a random variable which depends only on . So, for , are independent of . Now, if

    by Lemma (A.1), we have

    Since for all , we also have


    Pay a failure probability of and assume the conclusion of Lemma (A.2) and we have:

    Now, it is easy to see that increases as increases subject to (11). So,

    contradicting the definition of in the algorithm. This completes the proof of the Lemma.

  • (of Lemma A.4): After paying a failure probability of , assume no threshold splits any . [The factors of and come in because we are taking the union bound over all words and all topics.] Then,

    Wlg, assume that . Then, with probability, at least , . Now, either and all are zero and then , or , whence, . So, and . So,