Topic models  assume that each document in a text corpus is generated from an ad-mixture of topics, where, each topic is a distribution over words in a Vocabulary. An admixture is a convex combination of distributions. Words in the document are then picked in i.i.d. trials, each trial has a multinomial distribution over words given by the weighted combination of topic distributions. The problem of inference, recovering the topic distributions from such a collection of documents, is provably NP-hard. Existing literature pursues techniques such as variational methods  or MCMC procedures 
for approximating the maximum likelihood estimates.
Given the intractability of the problem one needs further assumptions on topics to derive polynomial time algorithms which can provably recover topics. A possible (strong) assumption is that each document has only one topic but the collection can have many topics. A document with only one topic is sometimes referred as a pure topic document.  proved that a natural algorithm, based on SVD, recovers topics when each document is pure and in addition, for each topic, there is a set of words, called primary words, whose total frequency in that topic is close to 1. More recently,  show using tensor methods that if the topic weights have Dirichlet distribution, we can learn the topic matrix. Note that while this allows non-pure documents, the Dirichlet distribution gives essentially uncorrelated topic weights.
In an interesting recent development [4, 5] gave the first provable algorithm which can recover topics from a corpus of documents drawn from admixtures, assuming separability. Topics are said to be separable if in every topic there exists at least one Anchor word. A word in a topic is said to be an an Anchor word for that topic if it has a high probability in that topic and zero probability in remaining topics. The requirement of high probability in a topic for a single word is unrealistic.
Topic distributions, such as those learnt in LDA, try to model the co-occurrence of a group of words which describes a theme. Keeping this in mind we introduce the notion of Catchwords. A group of words are called Catchwords of a topic, if each word occurs strictly more frequently in the topic than other topics and together they have high frequency. This is a much weaker assumption than separability. Furthermore we observe, empirically, that posterior topic weights assigned by LDA to a document often have the property that one of the weights is significantly higher than the rest. Motivated by this observation, which has not been exploited by topic modeling literature, we suggest a new assumption. It is natural to assume that in a text corpus, a document, even if it has multiple themes, will have an overarching dominant theme. In this paper we focus on document collections drawn from dominant admixtures. A document collection is said to be drawn from a dominant admixture if for every document, there is one topic whose weight is significantly higher than other topics and in addition, for every topic, there is a small fraction of documents which are nearly purely on that topic. The main contribution of the paper is to show that under these assumptions, our algorithm, which we call TSVD , indeed provably finds a good approximation in total error to the topic matrix. We prove a bound on the error of our approximation which does not grow with dictionary size , unlike  where the error grows linearly with .
will denote respectively, the number of words in the dictionary, number of topics and number of documents. are large, whereas, is to be thought of as much smaller. Let . For each topic, there is a fixed vector in giving the probability of each word in that topic. Let be the matrix with these vectors as its columns.
Documents are picked in i.i.d. trials. To pick document , one first picks a vector of topic weights according to a fixed distribution on . Let be the weighted combination of the topic vectors. Then the words of the document are picked in i.i.d. trials; each trial picks a word according to the multinomial distribution with as the probabilities. All that is given as data is the frequency of words in each document, namely, we are given the matrix , where Note that , where, the expectation is taken entry-wise.
In this paper we consider the problem of finding given .
2 Previous Results
In this section we review literature related to designing provable algorithms for topic models. For an overview of topic models we refer the reader to the excellent survey. Provable algorithms for recovering topic models was started by . Latent Semantic Indexing(LSI)  remains a successful method for retrieving similar documents by using SVD.  showed that one can recover from a collection of documents, with pure topics, by using SVD based procedure under the additional Primary Words Assumption.  showed that in the admixture case, if one assumes Dirichlet distribution for the topic weights, then, indeed, using tensor methods, one can learn to error provided some added assumptions on numerical parameters like condition number are satisfied.
 If every topic has an anchor word, there is a polynomial time algorithm that returns an such that with high probability,
where, is the condition number of , is the minimum expected weight of a topic and is the number of words in each document.
Note that the error grows linearly in the dictionary size , which is often large. Note also the dependence of on parameters , which is, and on , which is . If, say, the word “run” is an anchor word for the topic “baseball” and , then the requirement is that every 10 th word in a document on this topic is “run”. This seems too strong to be realistic. It would be more realistic to ask that a set of words like - “run”, “hit”, “score”, etc. together have frequency at least 0.1 which is what our catchwords assumption does.
3 Learning Topics from Dominant Admixtures
Informally, a document is said to be drawn from a Dominant Admixture if the document has one dominant topic. Besides its simplicity, we show empirical evidence from real corpora to demonstrate that topic dominance is a reasonable assumption. The Dominant Topic assumption is weaker than the Pure Topic assumption. More importantly SVD based procedures proposed by  will not apply. Inspired by the Primary words assumption we introduce the assumption that each topic has a set of Catchwords which individually occur more frequently in that topic than others. This is again a much weaker assumption than both Primary Words and Anchor Words assumptions and can be verified experimentally. In this section we establish that by applying SVD on a matrix, obtained by thresholding the word-document matrix, and subsequent means clustering can learn topics having Catchwords from a Dominant Admixture corpus.
3.1 Assumptions: Catchwords and Dominant admixtures
Let be non-negative reals satisfying:
Dominant topic Assumption (a) For , document has a dominant topic such that
(b)For each topic , there are at least documents in each of which topic has weight at least .
Catchwords Assumption: There are disjoint sets of words - such that with defined in (9)
Part (b.) of the Dominant Topic Assumption is in a sense necessary for “identifiability” - namely for the model to have a set of document vectors so that every document vector is in the convex hull of these vectors. The Catchwords assumption is natural to describe a theme as it tries to model a unique group of words which is likely to co-occur when a theme is expressed. This assumption is close to topics discovered by LDA like models, which try to model of co-occurence of words. If , then, the assumption (5) says . In fact if , we do not expect to see word (in topic ), so it cannot be called a catchword at all.
A slightly different (but equivalent) description of the model will be useful to keep in mind. What is fixed (not stochastic) are the matrices and the distribution of the weight matrix . To pick document , we can first pick the dominant topic in document and condition the distribution of on this being the dominant topic. One could instead also think of being picked from a mixture of distributions. Then, we let and pick the words of the document in i.i.d multinomial trials as before. We will assume that
where, is the probability of topic being dominant. This is only approximately valid, but the error is small enough that we can disregard it.
For , let be the probability that and and the corresponding “empirical probability”:
Note that is a real number, whereas,
is a random variable with. We need a technical assumption on the (which is weaker than unimodality).
No-Local-Min Assumption: We assume that does not have a local minimum, in the sense:
The justification for the this assumption is two-fold. First, generally, Zipf’s law kind of behaviour where the number of words plotted against relative frequencies declines as a power function has often been observed. Such a plot is monotonically decreasing and indeed satisfies our assumption. But for Catchwords, we do not expect this behaviour - indeed, we expect the curve to go up initially as the relative frequency increases, then reach a maximum and then decline. This is a unimodal function and also satisfies our assumption. Indeed, we have empirically observed, see EXPTS, that these are essentially the only two behaviours.
Relative sizes of parameters Before we close the section we discuss the values of the parameters are in order. Here,
is large. For asymptotic analysis, we can think of it as going to infinity.is also large and can be thought of as going to infinity. [In fact, if , then, intuitively, we see that there is no use of a corpus of more than constant size - since our model has i.i.d. documents, intuitively, the number of samples we need should depend mainly on ]. is (much) smaller, but need not be constant.
refers to a generic constant independent of ; its value may be different in different contexts.
3.2 The TSVD Algorithm
Existing SVD based procedures for clustering on raw word-document matrices fail because the spread of frequencies of a word within a topic is often more (at least not significantly less) than the gap between the word’s frequencies in two different topics. Hypothetically the frequency for the word run, in the topic Sports, may range from say 0.01 on up. But in other topics, it may range from 0 to 0.005 say. The success of the algorithm will lie on correctly identifying the dominant topics such as sports by identifying that the word run has occurred with high frequency. In this example, the gap (0.01-0.005) between Sports and other topics is less than the spread within Sports (1.0-0.01), so a 2-clustering approach (based on SVD) will split the topic Sports into two. While this is a toy example, note that if we threshold the frequencies at say 0.01, ideally, sports will be all above and the rest all below the threshold, making the succeeding job of clustering easy.
There are several issues in extending beyond the toy case. Data is not one-dimensional. We will use different thresholds for each word; word will have a threshold . Also, we have to compute . Ideally we would not like to split any , namely, we would like that for each and and each , either most have or most have . We will show that our threshold procedure indeed achieves this. One other nuance: to avoid conditioning, we split the data into two parts and , compute the thresholds using and actually do the thresholding on . We will assume that the intial had columns, so each part now has columns. Also, partitions the columns of as well as those of . The columns of thresholded matrix are then clustered, by a technique we call Project and Cluster, namely, we project the columns of to its dimensional SVD subspace and cluster in the projection. The projection before clustering has recently been proven  (see also ) to yield good starting cluster centers. The clustering so found is not yet satisfactory. We use the classic Lloyd’s -means algorithm proposed by . As we will show, the partition produced after clustering, of is close to the partition induced by the Dominant Topics, . Catchwords of topic are now (approximately) identified as the most frequently occurring words in documents in . Finally, we identify nearly pure documents in (approximately) as the documents in which the catchwords occur the most. Then we get an approximation to by averaging these nearly pure documents. We now describe the precise algorithm.
3.3 Topic recovery using Thresholded SVD
Threshold SVD based K-means (TSVD)
Threshold SVD based K-means (TSVD)
Randomly partition the columns of into two matrices and of columns each.
Compute Thresholds on For each , let be the highest value of such that ;
Do the thresholding on :
SVD Find the best rank approximation to .
Identify Dominant Topics
Project and Cluster Find (approximately) optimal means clustering of the columns of .
Lloyd’s Algorithm Using the clustering found in Step 4(a) as the starting clustering, apply Lloyd’s algorithm means algorithm to the columns of (, not ).
Let be the partition of corresponding to the clustering after Lloyd’s. //*Will prove that *//
For each , compute the th highest element of .
Let where, .
Find Topic Vectors Find the highest among all and return the average of these as our approximation to .
Main Theorem Under the Dominant Topic, Catchwords and No-Local-Min assumptions, the algorithm succeeds with high probability in finding an so that
A note on the sample complexity is in order. Notably, the dependence of on is best possible (namely ) within logarithmic factors, since, if we had fewer than documents, a topic which is dominant with probability only may have none of the documents in the collection. The dependence of on needs to be at least : to see this, note that we only assume that there are
nearly pure documents on each topic. Assuming we can find this set (the algorithm approximately does), their average has standard deviation of aboutin coordinate . If topic vector has entries, each of size , to get an approximation of to error , we need the per coordinate error to be at most which implies . Note that to get comparable error in , we need a quadratic dependence on .
There is a long sequence of Lemmas to prove the theorem. The Lemmas and the proofs are given in Appendix. The essence of the proof lies in proving that the clustering step correctly identifies the partition induced by the dominant topics. For this, we take advantage of a recent development on the means algorithm from  [see also ], where, it is shown that under a condition called the Proximity Condition, Lloyd’s
means algorithm starting with the centers provided by the SVD-based algorithm, correctly identifies almost all the documents’ dominant topics. We prove that indeed the Proximity Condition holds. This calls for machinery from Random Matrix theory (in particular bounds on singular values). We prove that the singular values of the thresholded word-document matrix are nicely bounded. Once the dominant topic of each document is identified, we are able to find the Catchwords for each topic. Now, we rely upon part (b.) of the Dominant Topic assumption : that is there is a small fraction of nearly Pure Topic-documents for each topic. The Catchwords help isolate the nearly pure-topic documents and hence find the topic vectors. The proofs are complicated by the fact that each step of the algorithm induces conditioning on the data- for example, after clustering, the document vectors in one cluster are not anymore independent.
4 Experimental Results
We compare the thresholded SVD based k-means (TSVD222Resources available at: http://mllab.csa.iisc.ernet.in/tsvd) algorithm 3.2 with the algorithms of , Recover-KL and Recover-L2, using the code made available by the authors333http://www.cs.nyu.edu/~halpern/files/anchor-word-recovery.zip. We first provide empirical support for the algorithm assumptions in Section 3.1, namely the dominant topic and the catchwords assumption. Then we show on 4 different semi-synthetic data that TSVD provides as good or better recovery of topics than the Recover algorithms. Finally on real-life datasets, we show that the algorithm performs as well as  in terms of perplexity and topic coherence.
TSVD parameters () are not known in advance for real corpus. We tested empirically for multiple settings and the following values gave the best performance. Thresholding parameters used were: , . For finding the catchwords, in step 5. For finding the topic vectors (step 6), taking the top 50% () gave empirically better results. The same values were used on all the datasets tested. The new algorithm is sensitive to the initialization of the first k-means step in the projected SVD space. To remedy this, we run 10 independent random initializations of the algorithm with K-Means++  and report the best result.
Datasets: We use four real word datasets in the experiments. As pre-processing steps we removed standard stop-words, selected the vocabulary size by term-frequency and removed documents with less than 20 words. Datasets used are: (1) NIPS444http://archive.ics.uci.edu/ml/datasets/Bag+of+Words: Consists of 1,500 NIPS full papers, vocabulary of 2,000 words and mean document length 1023. (2) NYT44footnotemark: 4: Consists of a random subset of 30,000 documents from the New York Times dataset, vocabulary of 5,000 words and mean document length 238. (3) Pubmed44footnotemark: 4: Consists of a random subset of 30,000 documents from the Pubmed abstracts dataset, vocabulary of 5,030 words and mean document length 58. (4) 20NewsGroup555http://qwone.com/~jason/20Newsgroups (20NG): Consist of 13,389 documents, vocabulary of 7,118 words and mean document length 160.
4.1 Algorithm Assumptions
To check the dominant topic and catchwords assumptions, we first run 1000 iterations of Gibbs sampling on the real corpus and learn the posterior document-topic distribution () for each document in the corpus (by averaging over 10 saved-states separated by 50 iterations after the 500 burn-in iterations). We will use this posterior document-topic distribution as the document generating distribution to check the two assumptions.
Dominant topic assumption: Table 1 shows the fraction of the documents in each corpus which satisfy this assumption with (minimum probability of dominant topic) and (maximum probability of non-dominant topics). The fraction of documents which have almost pure topics with highest topic weight at least 0.95 () is also shown. The results indicate that the dominant topic assumption is well justified (on average 64% documents satisfy the assumption) and there is also a substantial fraction of documents satisfying almost pure topic assumption.
Catchwords assumption: We first find a -clustering of the documents
by assigning all documents which have highest posterior probability for the same topic into one cluster. Then we use step 5 of TSVD (Algorithm3.2) to find the set of catchwords for each topic-cluster, i.e. , with the parameters: , (taking into account constraints in Section 3.1, ). Table 1 reports the fraction of topics with non-empty set of catchwords and the average per topic frequency of the catchwords666. Results indicate that most topics on real data contain catchwords (Table 1, second-last column). Moreover, the average per-topic frequency of the group of catchwords for that topic is also quite high (Table 1, last column).
No-Local-Min Assumption: To provide support and intuition for the local-min assumption we consider the quantity , in (7). Recall that , we will analyze the behavior of as a function of for some topics and words . As defined, we need a fixed to check this assumption and so we generate semi-synthetic data with a fixed from LDA model trained on the real NYT corpus (as explained in Section 4.2.1). We find catchwords and the sets as in the catchwords assumption above and plot separately for some random catchwords and non-catchwords by fixing some random . Figure 1 shows the plots. As explained in 3.1, the plots are monotonically decreasing for non-catchwords and satisfy the assumption. On the other hand, the plots for catchwords are almost unimodal and also satisfy the assumption.
|Corpus||% s with Dominant||% s with Pure||% Topics||CW Mean|
|Topics ()||Topics ()||with CW||Frequency|
4.2 Empirical Results
4.2.1 Topic Recovery on Semi-Synthetic Data
Semi-synthetic Data: Following , we generate semi-synthetic corpora from LDA model trained by MCMC, to ensure that the synthetic corpora retain the characteristics of real data. Gibbs sampling is run for 1000 iterations on all the four datasets and the final word-topic distribution is used to generate varying number of synthetic documents with document-topic distribution drawn from a symmetric Dirichlet with hyper-parameter 0.01. For NIPS, NYT and Pubmed we use topics, for 20NewsGroup , and mean document lengths of 1000, 300, 100 and 200 respectively. Note that the synthetic data is not guaranteed to satisfy dominant topic assumption for every document (on average about 80% documents satisfy the assumption for value of tested in Section 4.1)
Topic Recovery: We learn the word-topic distributions () for the semi-synthetic corpora using TSVD and the Recover algorithms of . Given these learned topic distributions and the original data-generating distributions (), we align the topics of and by bipartite matching and rearrange the columns of in accordance to the matching with . Topic recovery is measured by the average of the error across topics (called reconstruction error ), , defined as: .
We report reconstruction error in Table 2 for TSVD and the Recover algorithms, Recover-L2 and Recover-KL. TSVD has smaller error on most datasets than the Recover-KL algorithm. We observed performance of TSVD to be always better than Recover-L2. Best performance is observed on NIPS which has largest mean document length, indicating that larger leads to better recovery. Results on 20NG are slightly worse than Recover-KL for small sample size (though better than Recover-L2), but the difference is small. While the values in Table 2 are averaged values, Figure 2 shows that TSVD algorithm achieves much better topic recovery (27% improvement in error over Recover-KL) for majority of the topics (90%) on most datasets.
4.2.2 Topic Recovery on Real Data
A standard quantitative measure used to compare topic models and inference algorithms is perplexity . Perplexity of a set of test documents, where each document consists of words, denoted by , is defined as: . To evaluate perplexity on real data, the held-out sets consist of 350 documents for NIPS, 10000 documents for NYT and Pubmed, and 6780 documents for 20NewsGroup. Table 3 shows the results of perplexity on the 4 datasets. TSVD gives comparable perplexity with Recover-KL, results being slightly better on NYT and 20NewsGroup which are larger datasets with moderately high mean document lengths.
 proposed Topic Coherence as a measure of semantic quality of the learned topics by approximating user experience of topic quality on top words of a topic. Topic coherence is defined as , where is the document frequency of a word , is the document frequency of and together, and is a small constant. We evaluate TC for the top 5 words of the recovered topic distributions and report the average and standard deviation across topics. TSVD gives comparable results on Topic Coherence (see Table 3).
Topics on Real Data:
Table 4 shows the top 5 words of all 50 matched pair of topics on NYT dataset for TSVD, Recover-KL and Gibbs sampling. Most of the topics recovered by TSVD are more closer to Gibbs sampling topics. Indeed, the total average error with topics from Gibbs sampling for topics from TSVD is 0.034, whereas for Recover-KL it is 0.047 (on the NYT dataset).
|NIPS||754||749||835||-86.4 24.5||-88.6 22.7||-65.2 29.4|
|NYT||1579||1685||1555||-105.2 25.0||-102.1 28.2||-107.6 25.7|
|Pubmed||1188||1203||1307||-94.0 22.5||-94.4 22.5||-84.5 28.7|
|20NG||2431||2565||2390||-93.7 13.6||-89.4 20.7||-90.4 27.0|
Summary: We evaluated the proposed algorithm, TSVD, rigorously on multiple datasets with respect to the state of the art (Recover), following the evaluation methodology of . In Table 2 we show that the L1 reconstruction error for the new algorithm is small and on average 19.6% better than the best results of the Recover algorithms . We also demonstrate that on real datasets the algorithm achieves comparable perplexity and topic coherence to Recover (Table 3. Moreover, we show on multiple real datasets that the algorithm assumptions are well justified in practice.
Real world corpora often exhibits the property that in every document there is one topic dominantly present. A standard SVD based procedure will not be able to detect these topics, however TSVD, a thresholded SVD based procedure, as suggested in this paper, discovers these topics. While SVD is time-consuming, there have been a host of recent sampling-based approaches which make SVD easier to apply to massive corpora which may be distributed among many servers. We believe that apart from topic recovery, thresholded SVD can be applied even more broadly to similar problems, such as matrix factorization, and will be the basis for future research.
|zzz_elian zzz_miami boy father zzz_cuba||zzz_elian boy zzz_miami father family||zzz_elian zzz_miami boy father zzz_cuba|
|cup minutes add tablespoon oil||cup minutes tablespoon add oil||cup minutes add tablespoon oil|
|game team yard zzz_ram season||game team season play zzz_ram||team season game coach zzz_nfl|
|book find british sales retailer||book find school woman women||book find woman british school|
|run inning hit season game||run season game inning hit||run season game hit inning|
|church zzz_god religious jewish christian||pope church book jewish religious||religious church jewish jew zzz_god|
|patient drug doctor cancer medical||patient drug doctor percent found||patient doctor drug medical cancer|
|music song album musical band||black reporter zzz_new_york zzz_black show||music song album band musical|
|computer software system zzz_microsoft company||web www site cookie cookies||computer system software technology mail|
|house dog water hair look||room show look home house||room look water house hand|
|zzz_china trade zzz_united_states nuclear official||zzz_china zzz_taiwan government trade zzz_party||zzz_china zzz_united_states zzz_u_s zzz_clinton zzz_american|
|zzz_russian war rebel troop military||zzz_russian zzz_russia war zzz_vladimir_putin rebel||war military zzz_russian soldier troop|
|officer police case lawyer trial||zzz_ray_lewis police case officer death||police officer official case investigation|
|car driver wheel race vehicles||car driver truck system model||car driver truck vehicle wheel|
|show network zzz_abc zzz_nbc viewer||con zzz_mexico son federal mayor||show television network series zzz_abc|
|com question information zzz_eastern sport||com information question zzz_eastern sport||com information daily question zzz_eastern|
|book author writer com reader||zzz_john_rocker player team right braves||book word writer author wrote|
|zzz_al_gore zzz_bill_bradley campaign president democratic||zzz_al_gore zzz_bill_bradley campaign president percent||zzz_al_gore campaign zzz_bill_bradley president democratic|
|actor film play movie character||goal play team season game||film movie award actor zzz_oscar|
|school student teacher district program||school student program million children||school student teacher program children|
|tax taxes cut billion plan||zzz_governor_bush tax campaign taxes plan||tax plan billion million cut|
|percent stock market fund investor||million percent tax bond fund||stock market percent fund investor|
|team player season coach zzz_nfl||team season player coach zzz_cowboy||team player season coach league|
|family home friend room school||look gun game point shot||family home father son friend|
|primary zzz_mccain voter zzz_john_mccain zzz_bush||zzz_john_mccain zzz_george_bush campaign republican voter||zzz_john_mccain zzz_george_bush campaign zzz_bush zzz_mccain|
|zzz_microsoft court company case law||zzz_microsoft company computer system software||zzz_microsoft company window antitrust government|
|company million percent shares billion||million company stock percent shares||company million companies business market|
|site web sites com www||web site zzz_internet company com||web site zzz_internet online sites|
|scientist human cell study researcher||dog quick jump altered food||plant human food product scientist|
|baby mom percent home family||mate women bird film idea||women look com need telegram|
|point game half shot team||point game team season zzz_laker||game point team play season|
|zzz_russia zzz_vladimir_putin zzz_russian zzz_boris_yeltsin zzz_moscow||zzz_clinton government zzz_pakistan zzz_india zzz_united_states||government political election zzz_vladimir_putin zzz_russia|
|com zzz_canada www fax information||chocolate food wine flavor buy||www com hotel room tour|
|room restaurant building fish painting||zzz_kosovo police zzz_serb war official||building town area resident million|
|loved family show friend play||film show movie music book||film movie character play director|
|prices percent worker oil price||percent stock market economy prices||percent prices economy market oil|
|million test shares air president||air wind snow shower weather||water snow weather air scientist|
|zzz_clinton flag official federal zzz_white_house||zzz_bradley zzz_al_gore campaign zzz_gore zzz_clinton||zzz_clinton president gay mayor zzz_rudolph_giuliani|
|files article computer art ball||show film country right women||art artist painting museum show|
|con percent zzz_mexico federal official||official zzz_iraq government zzz_united_states oil||zzz_mexico drug government zzz_united_states mexican|
|involving book film case right||test women study student found||plane flight passenger pilot zzz_boeing|
|zzz_internet companies company business customer||company companies deal zzz_internet zzz_time_warner||media zzz_time_warner television newspaper cable|
|zzz_internet companies company business customer||newspaper zzz_chronicle zzz_examiner zzz_hearst million||million money worker company pay|
|goal play games king game||zzz_tiger_wood shot tournament tour player||zzz_tiger_wood tour tournament shot player|
|zzz_american zzz_united_states zzz_nato camp war||zzz_israel zzz_lebanon peace zzz_syria israeli||zzz_israel peace palestinian talk israeli|
|team season game player play||team game point season player||race won win fight team|
|reporter zzz_earl_caldwell zzz_black black look||corp group list oil meeting||black white zzz_black hispanic reporter|
|campaign zzz_republican republican zzz_party primary||zzz_bush zzz_mccain campaign republican voter||gun bill law zzz_congress legislation|
|zzz_bush zzz_mccain campaign primary republican||flag black zzz_confederate right group||flag zzz_confederate zzz_south_carolina black zzz_south|
|zzz_john_mccain campaign zzz_george_bush zzz_bush republican||official government case officer security||court law case lawyer right|
-  Blei, D. Introduction to probabilistic topic models. Communications of the ACM, pp. 77–84, 2012.
Blei, D., Ng, A., and Jordan, M.
Latent Dirichlet allocation.
Journal of Machine Learning Research, pp. 3:993–1022, 2003. Preliminary version in Neural Information Processing Systems 2001.
-  Griffiths, T. L. and Steyvers, M. Finding scientific topics. Proceedings of the National Academy of Sciences, 101:5228–5235, 2004.
-  Arora, S., Ge, R., and Moitra, A. Learning topic models – going beyond SVD. In Foundations of Computer Science, 2012b.
-  Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag, D., Wu, Y., and Zhu M. A practical algorithm for topic modeling with provable guarantees. In Internation Conference on Machine Learning, 2013
-  Anandkumar, A., Foster, D., Hsu, D., Kakade, S., and Liu, Y. A Spectral Algorithm for Latent Dirichlet Allocation In Neural Information Processing Systems, 2012.
-  Papadimitriou, C., Raghavan, P., Tamaki H., and Vempala S. Latent semantic indexing: a probabilistic analysis. Journal of Computer and System Sciences, pp. 217–235, 2000. Preliminary version in PODS 1998.
-  Deerwester, S., Dumais, S., Landauer, T., Furnas, G., and Harshman, R. Indexing by latent semantic analysis. Journal of the American Society for Information Science, pp. 391–407, 1990.
-  Kumar, A., and Kannan, R. Clustering with spectral norm and the k-means algorithm. In Foundations of Computer Science, 2010
-  Awashti, P., and Sheffet, O. Improved spectral-norm bounds for clustering. In Proceedings of Approx/Random, 2012.
Mimno, D., Wallach, H., Talley, E., Leenders, M. and McCallum, A.
Optimizing semantic coherence in topic models.
Empirical Methods in Natural Language Processing, pp. 262–272, 2011.
-  Lloyd, Stuart P. Least squares quantization in PCM, IEEE Transactions on Information Theory 28 (2): 129–137,1982.
-  Arthur, D., and Vassilvitskii, S. K-means++: The advantages of careful seeding. In Proceedings of ACM-SIAM symposium on Discrete algorithms, pp. 1027–1035, 2007
-  McDiarmid, C. On the method of Bounded Differences. Surveys in Combinatorics: London Math. Soc. Lecture Note Series 141. Cambridge University Press., 1989.
-  Vershynin R. Introduction to non-asymptotic analysis of random matrices. In ArXiv:1011.3027v6 [math.PR] 4 Oct 2011
Appendix A Line of Proof
We describe the Lemmas we prove to establish the result. The detailed proofs are in the Section B.
a.1 General Facts
We start with a consequence of the no-local-minimum assumption. We use that assumption solely through this Lemma.
Let be as in (6). If for some and , and also then, .
Next, we state a technical Lemma which is used repeatedly. It states that for every , the empirical probability that is close to the true probability. Unsurprisingly, we prove it using H-C. But we will state a consequence in the form we need in the sequel.
a.1.1 Properties of Thresholding
Say that a threshold “splits” if has a significant number of with and also a significant number of with . Intuitively, it would be desirable if no threshold splits any , so that, in , for each , either most have or most have . We now prove that this is indeed the case with proper bounds. We henceforth refer to the conclusion of the Lemma below by the mnemonic “no threshold splits any ”.
(No Threshold Splits any ) For a fixed , with probability at least , the following holds:
Let be a matrix whose columns are given by
’s columns corresponding to all are the same. The entries of the matrix are fixed (real numbers) once we have (and the thresholds are determined). Note: We have “integrated out ”, i.e.,
(So, think of for ’s columns being picked first from which is calculated. for columns of are not yet picked until the are determined.) But are random variables before we fix . The following Lemma is a direct consequence of “no threshold splits any ”.
Let . With probability at least (over the choice of ):
So far, we have proved that for every , the threshold does not split any . But this is not sufficient in itself to be able to cluster (and hence identify the ), since, for example, this alone does not rule out the extreme cases that for most in every , is above the threshold (whence for almost all ) or for most in no is above the threshold, whence, for almost all . Both these extreme cases would make us loose all the information about due to thresholding; this scenario and milder versions of it have to be proven not to occur. We do this by considering how thresholds handle catchwords. Indeed we will show that for a catchword , a has above the threshold and a has below the threshold. Both statements will only hold with high probability, of course and using this, we prove that and are not too close for in different ’s. For this, we need the following Lemmas.
For , and , we have with ,
With probability at least , we have
Next, we wish to show that clustering as in TSVD identifies the dominant topics correctly for most documents, i.e., that for all . For this, we will use a theorem from  [see also ] which in this context says:
If all but a fraction of the the satisfy the “proximity condition”, then TSVD identifies the dominant topic in all but fraction of the documents correctly after polynomial number of iterations.
To describe the proximity condition, first let be the maximum over all directions of the square root of the mean-squared distance of to , i.e.,
The parameter should remind the reader of standard deviation, which is indeed what this is, since . Our random variables being dimensional vectors, we take the maximum standard deviation in any direction.
is said to satisfy the proximity condition with respect to , if for each and each and and each and , the projection of onto the line joining and is closer to by at least
than it is to . [Here, is a constant.]
To prove proximity, we need to bound . This will be the task of the subsection B.1 which relies heavily on Random Matrix Theory.
Appendix B Proofs of Correctness
We start by recalling the Höffding-Chernoff (H-C) inequality in the form we use it.
Höffding-Chernoff If is the average of independent random variables with values in and , then, for an ,
(of Lemma A.1) Abbreviate by . We claim that either (i) or (ii) To see this, note that if both (i) and (ii) fail, we have and with . But then there has to be a local minimum of between and . If (i) holds, clearly, and so the lemma follows. So, also if (ii) holds.
(of Lemma A.2) Note that where, is the indicator variable of . and we apply H-C with and . We have , as is easily seen by calculating the roots of the quadratic . Thus we get the claimed for . Note that the same proof applies for as well as .
To prove the second assertion, let and , then, satisfies the quadratic inequalities:
By bounding the roots of these quadratics, it is easy to see the second assertion after some calculation.
(of Lemma A.3) Note that is a random variable which depends only on . So, for , are independent of . Now, if
by Lemma (A.1), we have
Since for all , we also have
Pay a failure probability of and assume the conclusion of Lemma (A.2) and we have:
Now, it is easy to see that increases as increases subject to (11). So,
contradicting the definition of in the algorithm. This completes the proof of the Lemma.
(of Lemma A.4): After paying a failure probability of , assume no threshold splits any . [The factors of and come in because we are taking the union bound over all words and all topics.] Then,
Wlg, assume that . Then, with probability, at least , . Now, either and all are zero and then , or , whence, . So,