1 Introduction
Democracy is a central variable in economics and in political science. Why are some countries more democratic than others? Why do coups happen? Does democracy impact economic policy? Does democratization affect the probability of a country going to war? These are some of the questions with which economists and political scientists concern themselves.
To answer those questions researchers need democracy to be measured somehow. That is where democracy indices come in. There are at least twelve democracy indices today (Pemstein, Meserve, and Melton 2010), the most popular of which is the Polity (Marshall, Gurr, and Jaggers 2013), which assigns a score from 10 to +10 to each of 167 countries in each year of the 19462013 period. Another popular democracy index is the Freedom House one (Freedom House 2013), which assigns a score from 1 to 7 to each of 195 countries in each year of the 19722013 period.
All democracy indices draw to some extent from Dahl’s (1972) conceptualization: democracy as a mixture of competition and participation. Yet none are replicable or provide adequate measures of uncertainty. All democracy indices we have today rely directly or indirectly on country experts checking boxes on questionnaires. We do not observe what boxes they are checking, or why; all we observe are the final scores. The process is opaque and at odds with the increasingly demanding standards of openness and replicability of the field. More importantly, opacity makes it easy for country experts to boost the scores of countries that adopt the ‘correct’ policies.
Coding rules help, but still leave too much open for interpretation. For instance, consider this excerpt from the Polity handbook (Marshall, Gurr, and Jaggers 2013): ‘If the regime bans all major rival parties but allows minor political parties to operate, it is coded here. However, these parties must have some degree of autonomy from the ruling party/faction and must represent a moderate ideological/philosophical, although not political, challenge to the incumbent regime.’ (73). How do we measure autonomy? Can we always observe it? What is ‘moderate’? Clearly it is not hard to smuggle ideological contraband into democracy scores.
Ideological biases, in turn, make empirical tests circular. If we find an association between democracy and some policy is that a genuine association or an artifact of human coders’ preferences regarding ? With the democracy measures we have today it is hard to know. Every time we regress the Polity scores or the Freedom House (Freedom House 2013) scores on some policy we may be regressing not on but on instead.
Another problem with existing indices is the lack of proper standard errors. The two most popular indices  the Polity and the Freedom House  only give us point estimates, without any measure of uncertainty. That prevents us from knowing, say, whether Uruguay (Polity score = 10) is really more democratic than Argentina (Polity score = 8) or whether the uncertainty of the measurement process is sufficient to make them statistically indistinguishable. In other words, we cannot do descriptive inference.
Moreover, without standard errors we cannot do causal inference when democracy is one of the regressors. As Treier and Jackman (2008) warn, ‘whenever democracy appears as an exploratory variable in empirical work, there is an (almost always ignored) errorsinvariables problem, potentially invalidating the substantive conclusions of these studies’ (203).
Only one (publicly available) measure has standard errors: the Unified Democracy Scores (UDS), created by Pemstein et al. (2010). To produce the UDS Pemstein et al. (2010) treated democracy as a latent variable and used a multirater ordinal probit model to extract that latent variable from twelve different democracy measures (among which the Polity and the Freedom House). The UDS comes with point estimates (posterior means) and confidence intervals (posterior quantiles).
The UDS is a big improvement on all other measures, but their standard errors are too large to be useful. 70% of the countries are all statistically indistinguishable from each other (in the year 2008  the last year in the UDS dataset at the moment of writing); pairs as diverse (regimewise) as Denmark and Suriname, Poland and Mali, or New Zealand and Mexico have overlapping confidence intervals.
Hence we need a new democracy index  one that is replicable and comes with standard errors small enough to actually distinguish between cases. In this paper I use natural language processing to create such measure.
2 Methods
The basic idea is simple. News articles on, say, North Korea or Cuba contain words like ‘censorship’ and ‘repression’ more often than news articles on Belgium or Australia. Hence news articles contain quantifiable regimerelated information that we can use to create a democracy index.
To produce the ADS I relied on supervised learning. I tried three different approaches, compared the results, and picked the approach that worked best. More specifically, I tried: a) a combination of Latent Semantic Analysis and treebased regression methods; b) a combination of Latent Dirichlet Allocation and treebased regression methods; and c) the Wordscores algorithm. The Wordscores algorithm outperformed the alternatives.
In this section I explain the corpus selection and the approaches I tried.
2.1 Corpus
I use a total of 6,043 news sources. These are all the news sources in English available on LexisNexis Academic, which is an online repository of journalistic content. The list includes American newspapers like The New York Times, USA Today, and The Washington Post; foreign newspapers like The Guardian and The Daily Telegraph; news agencies like Reuters, Agence France Presse (English edition), and Associated Press; and online sources like blogs and TV stations’ websites.
I use LexisNexis’ internal taxonomy to identify and select articles that contain regimerelated news. In particular, I choose all articles with one or more of the following tags: ‘human rights violations’ (a subtag of ‘crime, law enforcement and corrections’); ‘elections and politics’ (a subtag of ‘government and public administration’); ‘human rights’ (a subtag of ‘international relations and national security’); ‘human rights and civil liberties law’ (a subtag of ‘law and legal system’); and ‘censorship’ (a subtag of ‘society, social assistance and lifestyle’).
LexisNexis’ news database covers the period 1980present (though actual coverage varies by news source), so in principle the ADS could cover that period as well. In practice, however, LexisNexis does not provide search codes for countries that have ceased to exist, so we cannot reliably retrieve news articles on, say, the Soviet Union or East Germany (we could search by the country’s name but that yields unreliable results  think of Turkey, for instance). Hence I limit myself to the 19922012 interval.
That selection  i.e., regimerelated news, all countries that exist today, 19922012  results in a total of about 42 million articles (around 4 billion words total), which I then organize by countryyear. To help reduce spurious associations I remove proper nouns (that should help prevent, for instance, ‘Washington’ being associated with high levels of democracy just because the word appears frequently on news stories featuring a democratic country), in a probabilistic way (if all occurrences of the word are capitalized then that is probably a proper noun and therefore it is removed).
For each countryyear I merge all the corresponding news articles into a single document and transform it into a termfrequency vector. I then merge all vectors together in a big termfrequency matrix.
I try two variations of the same collection of news articles. The first variation, which I call corpus A, is just as described above, with no changes. It contains about 6.3 million unique words after proper nouns are removed probabilistically. The second variation, which I call corpus B, is corpus A minus: a) the 100 most frequent words in the English language; and b) any word does not appear more than once in any of the documents. Hence corpus B corpus A. Corpus B has about 2.3 million unique words.
2.2 Supervised learning
I adopt a supervised learning approach. In supervised learning we feed the machine a number of prescored cases  the training data. The machine then ‘learns’ from the training data. In text analysis that means learning how the frequency of each word or topic varies according to the document scores. For instance, the algorithm may learn that the word ‘censorship’ is more frequent the lower the democracy score of the document. Finally, the algorithm uses that knowledge to assign scores to all other cases  i.e., to the test data.
The period 19922012 gives us a total of 4,067 countryyears. I choose the year 1992 for the training data and extract the corresponding scores from the Unified Democracy Scores  UDS (Pemstein et al. 2010). The UDS have data on 184 countries for the year 1992. Hence we have 184 samples in the training data and 3,883 (4,067  184) samples in the test data. I select the year 1992 simply because it is the first year in our dataset. I select the UDS because it is an amalgamation of several other democracy scores, which reduces measurement noise.
2.3 Latent Semantic Analysis
Latent Semantic Analysis (LSA) is a method for extracting topics from texts [Landauer, Foltz, and Laham 1998]. More concretely, LSA tells us two things: a) which words ‘matter’ more for each topic; and b) which topics appear more in each document.
We start with a termfrequency matrix, , where rows represent terms, columns represent documents, and each entry is the frequency of term on document . In our case each column represents the set of all news articles about a given country published in a given year  for instance, France1995 or Colombia2003.
Next we apply the transformation. The of each entry is given by its termfrequency () multiplied by , where is the total number of documents and is the number of documents in which word appears (i.e., the word’s document frequency – ; the ratio thus gives us the inverse document frequency – ).
What the transformation does is increase the importance of the word the more it appears in the document but the less it appears in the whole corpus. Hence it helps us reduce the weights of inane words like ‘the’, ‘of’, etc and increase the weights of discriminant words (i.e., words that appear a lot but only in a few documents). For more details on see Manning, Raghavan, and Schütze (2008).
The next step is normalization. Here we have documents of widely different sizes, ranging from a few kilobytes (documents corresponding to small countries, like Andorra or San Marino, which rarely appear in the news) to 15 megabytes (documents corresponding to the United States, Russia, etc  countries that appear in the news all the time). Longer documents contain more unique words and have larger
values, which may skew the results (Manning et al. 2008). To avoid that we normalize the columns of the
matrix, transforming them into unit vectors.We will call our normalized matrix . Its dimensions are ( is the number of unique words in all documents and is the number of documents).
Now we are finally ready to run LSA. In broad strokes, LSA is the use of a particular matrix factorization algorithm  singular value decomposition (SVD)  to decompose a termfrequency matrix or some transformation thereof (like
or normalized ) and extract the word weights and topic scores. Let us break down LSA into each step.We start by deciding how many topics we want to extract  call it . There is no principled way to choose , but for large corpus the rule of thumb is something between 100 and 300 (Martin and Berry 2011).
To extract the desired topics we use SVD to decompose , as follows:
(1) 
where
is an orthogonal matrix (i.e.,
) whose columns are the leftsingular vectors of ; is a diagonal matrix whose nonzero entries are the singular values of ; and is an orthogonal matrix whose columns are the rightsingular vectors of [Martin and Berry 2011]. (Notation: throughout this paper for any matrix , is its transpose and is its conjugate transpose).There are several algorithms for computing SVD and the one I use is the one created by Halko, Martisson, and Tropp (2011), which is designed to handle large matrices.
Once we have decomposed we truncate , , and . We do that by keeping only the first columns of , the first rows and first columns of , and the first rows of . Let us call these truncated matrices , , and . The truncated matrices give us what we want. maps words onto topics: each entry gives us the weight (‘salience’) of word on topic . For instance, if we ran LSA on a collection of medical articles the resulting matrix might look like this:
The largest word weights (in absolute values) help us see what topics underlie the set of texts  in this example, diabetes (topic #1), heart diseases (topic #2), and cancer (topic #3).
Importantly, each topic contains weights for all words that appear in the entire corpus. For instance, topic #3 contains not only cancerrelated words but also all other words: insulin, mellitus, heart, etc. Hence all topics have exactly the same length (). What changes is the weight each topic assigns to each word. E.g., in topic #3 cancerrelated words have the largest weights  which is why we label topic #3 ‘cancer’.
In real life applications the topics are usually not so clearcut. Stopwords (‘the’, ‘of’, ‘did’, etc) often have large weights in at least some of the topics. Also, it is common for the top 20 or 50 words to be very similar across two or more topics. Finally, in real life applications with large corpora we usually extract a few hundred topics, not just three.
The product , in turn, maps topics onto documents: each entry gives us the weight (‘salience’) of topic on document . If we ran LSA on a collection of medical articles the resulting product might look like this:
Importantly, the topics extracted with LSA are ordered: the first topic  i.e., the first column of  captures more variation than the second column, the second column captures more variation than the third column, and so on. Thus if we run LSA with (say) and then with , the first 200 topics will be the same in both matrices. In other words, each topic is independent of all topics extracted after it.
That is all there is to LSA: we use SVD to decompose , truncate the resulting matrices, and extract from them the word weights and topic weights.
Here I expect that LSA will generate regimerelated topics  say, ‘elections’, ‘repression’, and so on  and also extraneous topics. I will then use treebased methods to create democracy scores, using all topics. Afterwards I should be able to inspect which topics are influencing the scores the most, drop the rows of corresponding to topics that are both extraneous and influential, and generate a new, improved set of democracy scores.
There is no clear rule for setting the number of topics. Here I set the number of topics () alternately to 50, 100, 150, to see how the results change. For corpus B I also try and (I cannot do that with corpus A because of memory limitations).
2.4 Latent Dirichlet Allocation
LSA, discussed in the previous section, is a flexible technique: it does not assume anything about how the words are generated. LSA is at bottom a data reduction technique; its core math  truncated SVD  works as well for mapping documents onto topics as it does for compressing image files.
But that flexibility comes at the cost of interpretability. The word weights and topic weights we get from LSA do not have a natural interpretation. We know that the word weights represent the ‘salience’ of each word for each topic, and that the topic weights represent the ‘salience’ of each topic on each document, but beyond that we cannot say much. ‘Salience’ does not have a natural interpretation in LSA.
That is the motivation for Latent Dirichlet Allocation (LDA), which was created by Blei, Ng, and Jordan (2003). Whereas LSA is modelfree, LDA models every aspect of the datagenerating process of the texts. We lose generality (the results are only as good as the assumed model), but we gain interpretability: with LDA we also get word weights and topic weights, but they have a clear meaning (more on this later).
It is unclear under what conditions LSA or LDA tends to produce superior results (Anaya 2011). The topics extracted by LDA are generally believed to be more clearcut than those extracted by LSA, but on the other hand they are also believed to be broader (Crain et al. 2012). Hence I try both LSA and LDA and compare the results.
Just as we did in LSA, here too we start by transforming our texts into data, i.e., into a termfrequency matrix, where each entry is the frequency of term on document .
Unlike what we did in LSA here we will use the term frequencies directly, without any or normalization (LDA models the datagenerating process of term frequencies, not of values or any other transformations).
LDA assumes the following datagenerating process for each document (here I draw heavily from Blei et al. 2003). We begin by choosing the number of words in the document, . We draw
from a Poisson distribution:
. Next we create a dimensional vector, , that contains the topic proportions in the document. For instance, if then 30% of the words will be assigned to the first topic, 20% to the second topic, and 50% to the third topic (to continue the example in the previous section these topics may be, say, ‘diabetes’, ‘heart diseases’, and ‘cancer’). We draw from a Dirichlet distribution: .Now that we have the number of words () and the topic distribution () we are ready to choose each word, . First we draw its topic, , from the topics in , as follows: . We then draw from , where is a matrix whose entry is the probability of word being selected if we randomly draw a word from topic ( is the total number of unique words in the corpus). For instance, the word ‘insulin’ may have probability 0.05 of being selected from the topic ‘diabetes’ and 0.001 of being selected from the topic ‘heart diseases’.
There are three levels in the model: and are the same for all documents, is specific to each document, and and are specific to each word in each document. We disregard because, as Blei et al. (2003) note, is an ancillary variable, independent of everything else in the model, so we can ignore its randomness.
That is all there is to the datagenerating model behind LDA. To estimate the model we need to find the and that maximize the probability of observing the s:
(2) 
That function is intractable, which precludes exact inference. We need to use some approximative algorithm. I use the one created by Hoffman, Blei, and Bach hoffman10, which is suitable for large matrices.
Like LSA, LDA also yields an matrix of wordsXtopics weights and a matrix of topicsXdocuments weights. Unlike LSA though, here these estimates have a natural interpretation. Each word weight is the probability of word being selected if we randomly draw a word from topic . And each topic weight is the proportion of words in document drawn from topic .
As with LSA here too I try different numbers of topics: 50, 100, and 150 for both corpora (A and B) and 200 and 300 for corpus B only.
2.5 Treebased regression techniques
In this section I motivate the use of treebased techniques and explain each in detail.
2.5.1 Motivation
LSA and LDA give us word weights, which map words onto topics, and topic weights, which map topics onto documents. But how do we go from that to democracy scores?
In principle we could use OLS. As explained before, our training samples are the 184 independent countries in the year 1992, whose scores we extract from the UDS, and our test samples are the 3,883 independent countries in the years 19932012. Thus we could, in principle: a) regress the topic scores of the 184 training samples on their respective UDS scores; and b) use the estimated coefficients to compute (‘predict’) the democracy scores of the 3,883 test samples.
But that would not work here. We have between 50 and 300 topics and only 184 reference cases, so with OLS we would quickly run out of degrees of freedom. Even with only 50 topics we would still be violating the ‘10 observations per variable’ rule of thumb. And there may be all sorts of interactions and other nonlinearities and to model these we would need additional terms, which would require even more degrees of freedom.
Thus I use treebased regression techniques instead. These techniques split the observations recursively, until they are all allocated in homogeneous ‘leaves’. Each split  and thus the path to each leaf  is based on certain values of the regressors (if then follow this branch, if then follow this other branch, etc). Treebased algorithms are nonparametric: we are not estimating any parameters, we are simply trying to find the best splitting points (i.e., the splitting points that make the leaves as homogeneous as possible).
Because treebased algorithms are nonparametric we can use them even if we have more variables than observations (Grömping 2009). Moreover, treebased algorithms handle nonlinearities well: the interrelations between variables are captured in the very hierarchical structure of the tree. We do not need to specify a priori what variables interact or in what ways. Hence my choice of treebased regression techniques over alternatives like LASSO, Ridge, or forward stepwise regression, all of which can handle a large number of variables but at the cost of ignoring nonlinearities.
There are several treebased methods and instead of choosing a particular one I try the four most prominent ones and compare the results. The next subsections explain each method in turn: decision trees, random forests, extreme random forests, and AdaBoost. In what follows I draw heavily from Hastie, Tibshirani, and Friedman (2008).
2.5.2 Decision trees
Decision trees were first introduced by Breiman et al. (1984). Say that we have a dependent variable , independent variables, and observations, and that all variables are continuous. We want to split the observations in two subsets (let us call them and ) based on some independent variable and some value . More specifically, we want to put all observations for which in one subset and all observations for which in another subset. But we do not want to choose and arbitrarily: we want to choose the and that make each subset as homogeneous as possible when it comes to . To do that we find the and that minimize the sum of the mean squared errors of the subsets:
(3) 
We find and iteratively. We then repeat the operation for the resulting subsets, partitioning each in two, and we keep doing so recursively until the subsets have fewer than observations. The lower the the better the model fits the training data but the worse it generalizes to the test data. There is no rigorous way to choose . Here I just follow two popular choices of ( and ) and compare the results.
The outcome is a decision tree that relates the independent variables to . For instance, if we tried to predict individual income based on socioeconomic variables our decision tree might look like this:
This is of course an extremely contrived example, but it gives us a concrete idea of what a decision tree looks like. Each node splits the observations in two groups according to some independent variable and some splitting point , chosen so as to minimize the sum of the mean squared errors of the two subsets immediately under that node. We stop growing the tree when the subsets become small enough  i.e., when the subsets contain fewer than observations. The very last subsets are the leaves of the tree. (The process can be tweaked in a number of ways  we can replace the mean squared error by other criteria, we can split the subsets in more than two, we can ‘prune’ the tree, etc , but here I stick to the basics.)
The average of each leaf gives us the predicted for new observations. Here, for instance, someone with 5 years of schooling, parental income over $50k, and age 30 would have a predicted income of $49.9k.
We can see how the tree captures nonlinearities. Parental income only matters when the individual has eight years of schooling or less. Height only matters when the individual has more than eight but twelve or less years of schooling. Age only matters for two groups of people: those with eight years of schooling or less and parental income over $50k; and those with more than twelve years of schooling. And so on.
With conventional regression we would need to model all such nonlinearities explicitly, positing a priori what depends on what. And we would need several interactive and higherorder terms, which would take up degrees of freedom. With decision tree learning, however, the nonlinearities are learned from the data, no matter how many or how complex they are (assuming of course that we have enough observations).
In real life applications the trees are usually much bigger, with hundreds or thousands of nodes and leaves.
2.5.3 Random forests
Random forests were first introduced in Breiman (2001). As the name suggests, random forests are an extension of decision trees. The idea is simple. We treat the reference set as a population, draw multiple bootstrap samples from it (with each sample having the same size as the population), and use each sample to grow a decision tree. To predict for new observations we simply average out the predicted s from the different trees.
The idea of random forests is to reduce noise. With a conventional decision tree small perturbations of the data can drastically impact the choice of and . By averaging out the predictions of multiple, bootstrapped trees we reduce that noise. Each bootstrapped tree yields poor predictions  slightly better than random guesses , but their average predictions should outperform those of a conventional tree.
There is no rigorous way to choose the number of bootstrapped trees, . Here I set . (I tried but the results were less stable  with the same data and parameters two different sets of of random forests with each would produce somewhat different results. With the results remain the same.)
The more the bootstrapped trees are different from each other, the more they will reduce noise in the end. Thus it is common to grow the trees in a slightly different way: instead of picking from all independent variables, we pick from a random subset of , with size (common choices are , and ). The smaller the , the more different the trees will be, and the more they will reduce noise in the end. But if the subset is too small (say, 1) each tree will perform so poorly that their combined performance will also be poor. Choosing the subset size is a matter of trial and error. I try , , and also simply , and compare the results.
I try and for each individual tree, just as before.
2.5.4 Extreme random forests
This is essentially the same as random forests, except that here we also randomize the choice of . Instead of finding the best for each we draw a random for each and then pick the best combination. This usually reduces noise a bit more (at the cost of degrading the performance of each individual tree).
As with random forests, ; ; and .
2.5.5 AdaBoost
AdaBoost (short for adaptive boosting) was first proposed by Freund and Schapire freund97. There are several variations thereof and the one I use here is Drucker’s (1997), popularly known as AdaBoost.R2, which I choose for being suitable to continuous outcomes (most AdaBoost variations are designed for categorical outcomes).
Say that we have observations. We start by choosing the number of models  in our case, trees  we want to create. We grow the first tree, , using all observations (unlike with random forest, where we use bootstrapped samples) and weighting each observation by , and compute each absolute error . We then find the largest error, , and use it to compute the adjusted error of every observation, . Next we calculate the adjusted error of the entire tree, . If we discard the tree and stop the process (if this happens with the very first tree that means AdaBoost failed). Otherwise we compute , update the observation weights ( is a normalizing constant), and grow the next tree, , using the updated weights. We repeat the process until all desired trees are grown or until . To predict for new observations we collect the individual predictions and take their weighted median, using as weights.
Intuitively, after each tree we increase the weights of the observations with the largest errors, then use the updated weights to grow the next tree. The goal is to force the learning process to concentrate on the hardest cases. Just as in random forests, here too we end up with multiple trees and we make predictions by aggregating the predictions of all trees. Unlike in random forests, however, here the trees are not independent and we aggregate their predictions by taking a weighted median rather than a simple mean.
As with random forests, (less when ); ; and .
2.6 Wordscores
The Wordscores algorithm was created by Laver, Benoit, and Garry (2003)  henceforth LBG. Unlike LSA or LDA it does not need to be combined with any regression method; it is a standalone algorithm that already gives us the scores of the test samples.
We begin by computing scores for each word. Let be the relative frequency of word on training document . The probability that we are reading document given that we see word is then . We let be the a priori score of training document and compute each word score as .
The second step is to use the word scores to compute the scores of the test documents (also called ‘virgin’ documents). Let be the relative frequency of word on virgin document . The score of virgin document is then . To score a virgin document we simply multiply each word score by its relative frequency and sum across.
The third step is the computation of uncertainty measures for the point estimates. LBG propose the following measure of uncertainty: , where and is the total number of virgin words. The term captures the dispersion of the word scores around the score of the document. Its square root divided by the square root of gives us a standard error, which we can use to assess whether two cases are statistically different from each other.
The fourth and final step is the rescaling of the test scores. In any given text the most frequent words are stopwords (‘the’, ‘of’, ‘and’, etc). Because stopwords have similar relative frequencies across all reference texts they will have centrist scores. That makes the scores of the virgin documents ‘bunch’ together around the middle of the scale; their dispersion is just not in the same metric as that of the training documents.
To correct for the ‘bunching’ of test scores LBG propose rescaling these as follows: , where is the raw score of virgin document , is the average raw score of all virgin documents,
is the standard deviation of the training scores, and
is the standard deviation of the virgin scores. This transformation expands the raw virgin scores by making them have the same standard deviation as the training scores. Martin and Vanberg (2008) propose an alternative rescaling formula, but Benoit and Laver (2008) show that the original formula is more appropriate when there are many test samples and few training samples, which is the case here.3 Results
I produced a total of 235 batches of democracy scores, each covering the same countryyears, i.e., all 3,883 countryyears in the 19932012 period. The first batch uses Wordscores and corpus A. The remaining 234 batches vary by corpus (A or B), topicextraction method (LSA or LDA), prior if the topicextraction method is LDA (symmetric or asymmetric normalized ), number of topics (50, 100, 150, 200, or 300), tree method (decision trees, random forests, extreme random forests, or AdaBoost), size of the subset of when splitting the tree nodes (, , or ), and minimum node size ( or ).
The correlation between the batch based on Wordscores and corpus A with the UDS is 0.74. Table 3 below shows the correlations between the UDS and some of the other 234 batches. The choices of corpus, , , and did not make much difference, so Table 3 only shows the correlations obtained with corpus B, , , and symmetric . The higher the correlation, the better the performance. (Replication material and instructions can be found at thiagomarzagao.com/papers)
As we observe, LSA and LDA perform poorly, with the correlations usually in the 0.100.20 range  way below the correlation of 0.74 obtained with Wordscores. The highest correlation obtained with LSA and LDA is 0.27 (in absolute value), which still implies an unacceptably high noisetosignal ratio.
I inspected every topic of every LSA and LDA specification. Either the words are disparate and do not form a coherent topic or the topic is too broad or not regimerelated. My initial idea was to inspect the most influential topics so I could know exactly what aspects of democracy are driving the results  and drop extraneous topics if necessary. But that is not feasible: the topics do not correspond to aspects of democracy. In a sense, all topics are extraneous. Hence the democracy scores produced with LSA and LDA are all but noise, which is why they correlate so weakly with the UDS.
The Wordscores results are clearly the best, so I inspect them more closely in the next section. I call the Wordscores results Automated Democracy Scores (ADS).
4 Overview of the ADS
The full 19932012 dataset is available for download (https://s3.amazonaws.com/thiagomarzagao/ADS.csv). Figure 2 below gives an idea of the ADS distribution in 2012.
As expected, democracy is highest in Western Europe and in the developed portion of the Englishspeaking world, and lowest in Africa and in the Middle East.
Figure 3 below shows that the ADS follow a normal distribution.
Table 4 below shows the ADS summary statistics by year.
As expected, the average ADS increases over time, from 0.006 in 1993 to 0.495 in 2012. That reflects the several democratization processes that happened over that period. We observe the same change in other democracy indices as well (between 1993 and 2012 the average Polity score (polity2) increased from 2.24 to 4.06 and the average Freedom House score (civil liberties + political rights) decreased from 7.46 to 6.63 (Freedom House scores decrease with democracy); the average UDS score increased from 0.21 to 0.41 between 1993 and 2008, the last year in the UDS dataset).
Also as expected, the standard errors decrease with press coverage. The larger the document with the countryyear’s news articles, the narrower the corresponding confidence interval. As Figure 4 shows, that relationship is not linear though: after 500KB or so the confidence intervals shrink dramatically and do not change much afterwards, not even when the document has 15MB or more.
4.1 The ADS vs other indices  point estimates
The ADS point estimates correlate 0.7439 with the UDS’ (posterior means), 0.6693 with the Polity’s (polity2), and 0.7380 with the Freedom House’s (civil liberties + political rights). Table 5 below breaks down these correlations by year.
As we see, the correlations do not vary much over time. This is a good sign: it means that the ADS are not overly influenced by the idiosyncrasies of the year 1992, from which we extract the training samples. Otherwise we would see the correlations decline sharply after 1993. The correlations do not vary much across indices either, other than being somewhat weaker for the Polity data. This is also a good sign: it means that the ADS are not overly influenced by the idiosyncrasies of the UDS, from which we extract the training scores. (Though we must remember that the UDS are partly based on the Polity and the Freedom House, so by extension the ADS also are.)
I also ran the algorithm using other years (rather than 1992) for the training data, using UDS as well. I also ran the algorithm using multiple years (up to all years but one) for the training data, again using UDS. Finally, I also ran the algorithm using not the UDS but the Polity and Freedom House indices for the training data. In all these scenarios the correlations remained in the vicinity of 0.70. This corroborates Klemmensen, Hobolt, and Hansen’s (2007) finding that Wordscores’ results are robust to the choice of training data.
(The samples we use for the training data cannot be used for the test data. For instance, in one scenario I used every other year for the training data, starting with 1992. In that scenario the training data was thus [1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012] and the test data was [1993 1995 1997 1999 2001 2003 2005 2007 2009 2011]. To compute the correlations with other indices I only used the test data.)
Countrywise, what are the most notable differences between the ADS and the UDS? Table 6 below shows the largest discrepancies.
The largest positive differences  i.e., the cases where the ADS are higher than the UDS  are mostly found in small countries with little press coverage. That is as expected: the less press attention, the fewer news articles we have to go by, and the harder it is to pinpoint the country’s ‘true’ democracy level.
The largest negative differences, however, tell a different story. It seems as if either the ADS repeatedly underestimate Israel’s democracy score or the UDS repeatedly overestimate it (and not only for the years shown in Table 6). We do not observe a country’s ‘true’ level of democracy, so we cannot know for sure whether the ADS or the UDS are biased (though of course these two possibilities are not mutually exclusive) but the ADS should be unbiased to the extent that we managed to filter out news articles not related to political regime; whatever biases exist in the UDS should become, by and large, random noise in the ADS.
For instance, imagine that the UDS are biased in favor of countries with generous welfare, like Sweden. The UDS of these countries will be ‘boosted’ somewhat. But to the extent that the news articles we selected are focused on political regime and not on welfare policy, Wordscores will not associate those boosted scores with welfarerelated words and hence the ADS will not be biased. The ADS will be less efficient, as (ideally) no particular words will be associated with those boosted scores, but that is it.
The UDS, on the other hand, rely on the assumption that ‘raters perceive democracy levels in a noisy but unbiased fashion’ (Pemstein et al. 2010:10), which as Bollen and Paxton (2000) have shown is simply not true. Hence whatever biases exist in the Polity, Freedom House, etc, wind up in the UDS as well. The datagenerating process behind the UDS does not mitigate bias in any way.
In other words, it seems more likely that the UDS are overestimating Israel’s democracy scores than that the ADS are underestimating them. This proIsrael bias is interesting in itself, but it also raises the more general question of whether the UDS might have an overall conservative bias. To investigate that possibility I performed a differenceofmeans test, splitting the data in two groups: countryyears with leftwing governments and countryyears with rightwing governments (I used the EXECLRC variable from Keefer’s (2002) Dataset of Political Institutions for data on government ideological orientation.)
The test rejected the null hypothesis that the mean ADSUDS difference is the same for the two groups: the mean ADSUDS difference for leftwing countryyears (0.127, std. error = 0.024, n = 802) is statistically smaller than the mean ADSUDS difference for rightwing countryyears (0.328, std. error = 0.025, n = 603), with p
0.00001. As both means are negative, it seems that the UDS tend to reward rightwing governments.I also checked whether the UDS may be biased toward economic policy specifically. I split the countryyears in the Index of Economic Freedom (Heritage Foun dation 2014) dataset into two groups: statist (IEF score below the median) and nonstatist (IEF score above the median). The differenceofmeans test shows that the mean ADSUDS difference for statists (0.132, std. error = 0.0196, n = 1057) is statistically lower than that of nonstatists (0.215, std. error = 0.015, n = 1977), with p 0.0006. Both means are negative here as well, so it seems that the UDS somehow reward free market policies.
We cannot conclusively indict the UDS or its constituent indices though. Perhaps democracy and rightwing government are positively associated and the ADS are somehow less efficient at capturing that association. This is consistent with the HayekFriedman hypothesis that leftwing governments are detrimental to democracy because economic activism expands the state’s coercive resources (Hayek 1944; Friedman 1962). As we do not observe a country’s true level of democracy, it is hard to know for sure what is going on here.
At least until we know whether the UDS are biased or the ADS are inefficient, the ADS are the conservative choice. Say we regress economic policy on the UDS and find that more democratic countries tend to have less regulation. Is that relationship genuine or is it an artifact of the UDS being biased in favor of free market policies? With biased measures our tests become circular: we cannot know the effect of on when our measure of is partly based on . Inefficiency, on the other hand, merely makes our tests more conservative.
4.2 The ADS vs other indices  standard errors
The ADS have much smaller standard errors than the UDS (the only other democracy index that also comes with standard errors). On average, each country in the ADS dataset in the year 2008 overlaps with other 4.49 countries; in the UDS dataset that average is 99.67. The ADS confidence intervals tend to be larger the less press coverage the country gets, but in all cases they are smaller than the corresponding UDS ones.
For instance, in the UDS the United States is statistically indistinguishable from 80 other countries, whereas in the ADS the United States is statistically indistinguishable from only one other country (Solomon Islands, which rarely appears in the news and thus has a wide confidence interval). The country with most overlaps in the UDS data is Sao Tome and Principe, which is statistically indistinguishable from 135 other countries. That makes 70% of the UDS scores (for 2008) statistically the same. The worst case in the ADS is Czech Republic, which overlaps with 25 other countries (in the UDS Czech Republic overlaps with 110 other countries).
The reason why the ADS standard errors are much smaller than the UDS ones is the sheer size of the data. We have 42 million news articles, which give us about 4 billion words in total. Because the total number of virgin words goes in the denominator of the formula for the standard errors, those 4 billion words shrink the confidence intervals dramatically.
The ADS standard errors also tell us something about the nature of democracy. The large standard errors of the UDS data might lead us to believe that democracy is better modeled as a categorical variable, like the one in Alvarez et al. (1996). Gugiu and Centellas (2013) claim that that is indeed the case: they use hierarchical cluster analysis to extract the latent democracy variable behind five existing indices (among which the Polity and Freedom House indices) and find that that latent variable is categorical, not continuous.
That conclusion is unwarranted though. If the constituent measures (Polity, Freedom House, etc) are too coarse to capture finegrained regime differences then it is not surprising that their latent variable will also be too coarse to capture finegrained regime differences. But just because a given measure fails to capture subtle distinctions does not mean that these distinctions do not exist. As the ADS standard errors suggest, these subtle distinctions do seem to exist.
5 Conclusion
The ADS address important limitations of the democracy indices we have today. The ADS are replicable and have standard errors narrow enough to distinguish cases. The ADS are also costeffective: all we need are training documents and training scores, both of which already exist; there is no need to hire dozens of country experts and spend months collecting and reviewing their work.
To facilitate replicability and extensibility I created a web application where anyone can tweak the training data and see how the results change: democracyscores.org.
It would be interesting to replicate existing (substantive) work on democracy but using the ADS instead, to see how the results change. The ADS come with standard errors, so we could incorporate these in the regressions, perhaps using errorsinvariables models (1987).
We could extend the method here to produce a daily or realtime democracy index. Existing indices are yearbased and outdated by 112 months, so we do not know how democratic a country is today or how democratic it was, say, on 11/16/2006. Automated text analysis can help us overcome those limitations. We cannot score the news articles from only one or two days, as there would not be enough data to produce meaningful results, but we can pick, say, the 12month period immediately preceding a certain date  for instance, 11/17/200511/16/2006 if we want democracy scores for 11/16/2006.
References

[Alvarez et al. 1996]
Alvarez, Mike, Cheibub, José, Limongi, Fernando, and Przeworski, Adam. 1996. Classifying political regimes.
Studies in Comparative International Development 31(2): 336.  [Anaya 2011] Anaya, Leticia. 2011. Comparing Latent Dirichlet Allocation and Latent Semantic Analysis as classifiers PhD dissertation, Department of Information and Decision Sciences, University of North Texas.
 [Benoit and Laver 2008] Benoit, Kenneth, and Laver, Michael. 2008. Compared to what? A comment on ’A robust transformation procedure for interpreting political text’ by Martin and Vanberg. Political Analysis 16(1): 10111.

[Blei, Ng, and Jordan 2003]
Blei, David, Ng, Andrew, and Jordan, Michael. 2003. Latent Dirichlet Allocation
Journal of Machine Learning Research
3: 9931022.  [Bollen and Paxton 2000] Bollen, Kenneth, and Paxton, Pamela. 2000. Subjective measures of liberal democracy. Comparative Political Studies 33(1): 5886.
 [Breiman et al. 1984] Breiman, Leo, Friedman, Jerome, Stone, Charles, and Ohlshen, R. A. 1984. Classification and regression trees. Wadsworth.
 [Breiman 2001] Breiman, Leo. 2001. Random forests. Machine Learning 45(1): 532.
 [Crain et al. 2012] Crain, Steven, Zhou, Ke, Yang, ShuangHong, and Zha Hongyuan. 2012. Dimensionalityreduction and topic modeling: from Latent Semantic Indexing to Latent Dirichlet Allocation and beyond. In Mining text data, eds. Aggarwal, Charu and Zhai, ChengXiang, 12961. Springer.
 [Dahl 1972] Dahl, Robert. 1972. Polyarchy. Yale University.
 [Drucker 1997] Drucker, Harris. 1997. Improving regressors using boosting techniques. ICML 97: 10715.
 [Freedom House 2013] Freedom House. 2013. Freedom in the world. Freedom House.
 [Freund and Schapire 1997] Freund, Yoav and Schapire, Robert. 1997. A decisiontheoretic generalization of online learning and an application to boosting. Journal of Computer and System Sciences 55(1): 11939.
 [Friedman 1962] Friedman, Milton. 1962. Capitalism and freedom. University of Chicago.
 [Fuller 1987] Fuller, Wayne. 1987. Measurement error models. John Wiley & Sons.

[Grömping 2009]
Grömping, Ulrike. 2009. Variable importance assessment in regression: linear regression versus random forest.
The American Statistician 63(4): 30919.  [Gugiu and Centellas 2013] Gugiu, Mihaeila, and Centellas, Miguel. 2013. The democracy cluster classification index. Political Analysis 21(3): 33449.
 [Halko, Martinsson, and Tropp 2011] Halko, Nathan, Martinsson, PerGunnar, and Tropp, Joel. 2011. Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review 53(2): 21788.
 [Hastie, Tibshirani, and Friedman 2008] Hastie, Trevor, Tibshirani, Robert, and Friedman, Jerome. 2008. The elements of statistical learning: data mining, inference, and prediction. Springer.
 [Hayek 1944] Hayek, Friedrich von. 1944. The road to serfdom. University of Chicago.
 [Heritage Foundation 2014] Heritage Foundation. 2014. Index of Economic Freedom Available at heritage.org/index
 [Hoffman, Blei, and Bach 2010] Hoffman, Matthew, Blei, David, and Bach, Francis. 2010. Online learning for Latent Dirichlet Allocation. NIPS 2(3): 19.
 [Keefer 2002] Keefer, Philip. 2002. DPI2000 Database of Political Institutions: changes and variable definitions Development Research Group, The World Bank.
 [Klemmensen, Hobolt, and Hansen 2007] Klemmensen, Robert, Hobolt, Sara, and Hansen, Martin. 2007. Estimating policy positions using political texts: an evaluation of the Wordscores approach. Electoral Studies 26(4): 74655.
 [Landauer, Foltz, and Laham 1998] Landauer, Thomas, Foltz, Peter, and Laham, Darrell. 1998. An introduction to Latent Semantic Analysis 25(23): 25984.
 [Laver, Benoit, and Garry 2003] Laver, Michael, Benoit, Kenneth, and Garry, John. 2003. Extracting policy positions from political texts using words as data. American Political Science Review, 97(2): 31131.
 [Manning, Raghavan, and Schütze 2008] Manning, Christopher, Raghavan, Prabhakar, and Schütze, Hinrich. 2008. Introduction to information retrieval. Cambridge University.
 [Marshall, Gurr, and Jaggers 2013] Marshall, Monty, Gurr, Ted, and Jaggers, Keith. 2013. Polity IV project: political regime characteristics and transitions, 18002012, dataset user’s manual. Center for Systemic Peace.
 [Martin and Berry 2011] Martin, Dian, and Berry, Michael. 2011. Mathematical foundations behind Latent Semantic Analysis. In Handbook of Latent Semantic Analysis, eds. Landauer, Thomas, McNamara, Danielle, Dennis, Simon, and Kintsch, Walter, 3555. Routledge.
 [Martin and Vanberg 2008] Martin, Lanny, and Vanberg, Georg. 2008. A robust transformation procedure for interpreting political text. Political Analysis 16(1): 93100.
 [Pemstein, Meserve, and Melton 2010] Pemstein, Daniel, Meserve, Stephen and Melton, James. 2010. Democratic compromise: a latent variable analysis of ten measures of regime type. Political Analysis 18(4): 42649.
 [Stewart 1993] Stewart, Gilbert. 1993. On the early history of the singular value decomposition. SIAM Review 35(4): 55166.
 [Treier and Jackman 2008] Treier, Shawn, and Jackman, Simon. 2008. Democracy as a latent variable. American Journal of Political Science 52(1): 20117.
Comments
There are no comments yet.