On Gobbledygook and Mood of the Philippine Senate: An Exploratory Study on the Readability and Sentiment of Selected Philippine Senators' Microposts

08/06/2015 ∙ by Fatima M. Moncada, et al. ∙ 0

This paper presents the findings of a readability assessment and sentiment analysis of selected six Philippine senators' microposts over the popular Twitter microblog. Using the Simple Measure of Gobbledygook (SMOG), tweets of Senators Cayetano, Defensor-Santiago, Pangilinan, Marcos, Guingona, and Escudero were assessed. A sentiment analysis was also done to determine the polarity of the senators' respective microposts. Results showed that on the average, the six senators are tweeting at an eight to ten SMOG level. This means that, at least a sixth grader will be able to understand the senators' tweets. Moreover, their tweets are mostly neutral and their sentiments vary in unison at some period of time. This could mean that a senator's tweet sentiment is affected by specific Philippine-based events.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Readability refers to a certain class of people’s perception of a text’s compellingness and requisite comprehensibility (McLaughlin, 1968). The degree of a text’s compellingness can be measured by determining the proportion of a certain class of people who read the text by choice. People who belong to the same class are those with closely similar Terminal Educational Age (Abrams, 1963). A person is compelled to a text  she is reading if she understands . Therefore, comprehensibility is a requisite to compellingness. A text’s measure of readability is ultimately dependent on a text’s linguistic characteristics (McLaughlin, 1968). Texts which are full of medical jargon, for example, have high readability scores among medical doctors but might have low readability scores among lawyers. This might have been one of the reasons why senator-judge Rodolfo G. Biazon of the Philippine Senate’s 11th Congress, a trained military man being the former Chief of Staff of the Armed Forces of the Philippines, stated the following during the trial of the impeachment case against former Philippine President Joseph Ejercito Estrada:

“… because of the legal exchanges that I could hardly put together, I said ‘let’s do away with this legal gobbledygook…”

This seemingly sarcarstic retort, once a favorite topic of conversation in the news media and among personalities in the broadcast version, was the result of a series of exchanges of lengthly legalese arguments among lawyers for and against President Estrada. The senator’s witty statement was meant to remove “legalistic gobbledygook” from “intelligent communication” within the Senate chamber that is composed of members who were nationally elected to represent, not only certain classes of people, but all Filipino people.

Given a text , how can one put a value on the readability of ? Word and sentence lengths are the linguistic characteristics that best predict a text’s reading difficulty (McLaughlin, 1968, 1969) or comprehensibility. Thus, by measuring word and sentence lengths, one can determine a text’s readability for a certain class of people, given that the text has been proven to be compelling for them (McLaughlin, 1968). Not until the use of legal gobbledygook was minimized in the proceedings of the Estrada impeachment that the Filipino people, already compelled to follow the proceedings because of the apparent human drama involving the highest official of the land, was able to totally comprehend it.

1.1 Simplified Measure of Gobbledygook

The Simple Measure of Gobbledygook or SMOG is a readability formula which calculates the approximate number of years of education required for a person to comprehend a given text. This formula is called the SMOG Grade, a function directly proportional to the total number of polysyllabic words in a text (McLaughlin, 1968). In computing for the SMOG Grade, a total of 30 sentences must be sampled from the text: ten consecutive sentences at the beginning, ten at the middle, and ten at the end. All words with more than two syllables must be counted to get the total number of polysyllables. Abbreviated words must be read as unabbreviated and numerical characters must be spelled out as well (McLaughlin, 1969). The SMOG Grade of a text  is shown as Equation 1, where  is the number of polysyllabic words in , and  is the number of sentences in :

(1)

It was experimentally found out that the SMOG precise formula yields a correlation coefficient of 0.985 and a standard error of 1.5159 

(McLaughlin, 1969). From the precise formula given above, a simpler equation  can be derived (Equation 2):

(2)

Although the simplified SMOG formula is less accurate, it is more preferred especially in fieldworks (McLaughlin, 1969). Its simple implementation and speed of use while still providing a rigorous method of measuring readability are what compelled researchers to use it instead of Equation 1 (McLaughlin, 1969).

1.2 SMOG in Readability Assessment of Health Messages

Because of SMOG’s usefulness in accessing communication materials from a certain class of experts to a more general class of people, it has since been used in different scientific researches that aim to assess the readability of different health communication materials (Brandt et al., 2005; Auta et al., 2011a, b; Gill et al., 2012; Wallace et al., 2010; Svider et al., 2013; Ache and Wallace, 2012; Wilson, 2009) and other health-related documents (Coyne et al., 2003). While most researches deal with printed communication materials, some studies assessed the readability of various health-related information available online (Edmunds et al., 2014; Stossel et al., 2012; Hoppe, 2010; Walsh and Volsko, 2008; Cherla et al., 2012).

In general, there are two approaches used in readability assessment using SMOG. First, the SMOG Grade is used to determine which grade level will be able to understand various educational and communication materials that are targeted towards a general audience (Brandt et al., 2005; Auta et al., 2011a; Gill et al., 2012; Makosky et al., 2009). Another approach used in readability researches is zeroing in to a particular audience group and determining, through the use of the SMOG formula, whether the communication materials can be understood easily by the specified target audience.

The latter approach is more commonly used in researches that aim to design, develop, test, and modify health messages (Makosky et al., 2009; Neuhauser et al., 2009; Holt et al., 2010; Patel and Simpson, 2010) or when the communication material being assessed was originally designed for a specific target audience (Swartz, 2010). For example, in developing printed educational materials on prostate cancer for church-attending African-American men, Holt et al(Holt et al., 2010) used the SMOG formula to assess their original materials and then revise them to a desirable level of sixth-grade reading difficulty. On the other hand, Swartz (Swartz, 2010) examined the readability of handouts and brochures on pediatric otitis media targeted towards parents. He determined whether the obtained SMOG Grade of eight corresponds to the reading capability of the publications’ intended audience. In addition, he also explored the correlation between the SMOG Grade and the parents’ actual reading satisfaction.

Aside from readability assessments of different health messages, SMOG has also been used as a tool in determining the effectiveness of semantic and syntactic text simplification. Nowadays, text simplification is done automatically through natural language processing, specifically using synonym generation and explanation generation. In analyzing whether the simplified text is indeed more readable than the original text, the SMOG formula is often used. If a simplified text scored lower than the original text in terms of SMOG, then the automated text simplification is considered effective 

(Kandula et al., 2010). However, Leroy et al(Leroy et al., 2013) noted that text simplification based on SMOG and other readability tests often results to more difficult text because these readability tests are focused on the writing style (i.e. word and sentence length) rather than the content itself. Therefore, it is of key importance that when using SMOG, careful interpretation and/or conclusions must be made in accordance to the tool’s limitations.

1.3 SMOG and Other Readability Tests

SMOG is often used in combination with other readability tests such as the Flesch-Kincaid Grade Level, Flesch Reading Ease, Fry Readability Formula, and the Gunning Fog Index. For example, Gill et al.c̃itep7 assessed the readability of publications released by the United States Center for Disease Control and Prevention on concussion and traumatic brain injury using SMOG and three other different readability tests. The materials’ Gunning Fog Index and Flesch-Kincaid Grade Level varied very closely at 11.1 and 11.3 respectively, with a Flesch Reading Ease index of 49.5. Interestingly, the computed SMOG grade for the tested materials was 12.8, notably higher than the two other tests. Another study which assessed the readability of patient education materials produced for the low-income population of the United States yielded similar results, where the SMOG grade (9.89) was significantly higher than the Flesch-Kincaid Grade (7.01) (Wilson, 2009). Consistent to such pattern, readability assessment of different medicine information in two separate studies (Auta et al., 2011b; Wallace et al., 2010) resulted to SMOG Grades that were higher than the Flesch-Kincaid Grade by 1 to 3 levels. In studies where the objects of readability assessment are Internet-based or online health information (Edmunds et al., 2014; Stossel et al., 2012; Walsh and Volsko, 2008; Cherla et al., 2012), the SMOG Grades remained significantly higher than their corresponding Flesch-Kincaid Grades; while, the Gunning FOG indexes are either equal to or slightly higher than the SMOG Grade.

When the SMOG formula is used with other readability tests and the results vary, some researchers would interpret the results collectively. For example, in assessing the readability of online resources on Graves’ disease and thyroid-associated ophthalmopathy (Edmunds et al., 2014), the US Department of Human and Health Sciences (USDHHS) standards for reading difficulty was used in interpreting the varying indexes obtained for SMOG, Flesch-Kincaid, Gunning-Fog, and Flesch Reading Ease. Following the recommended readability level (4 to 6) for online materials by the USDHHS, the study concluded that the online resources that were analyzed are too difficult for its audience to understand, with readability indixes of 11 for the Flesch-Kincaid formula, 13 for the SMOG and the Gunning-Fog formula, and 46 for the Flesch Reading Ease formula.

On the other hand, the Centers for Medicare and Medicaid Services or CMS recommends the use of SMOG as a standard in making written materials effective and clear to its audience (Stossel et al., 2012). Hence, other researchers opt to consider just the SMOG results when significant differences among the readability indexes are encountered. For example, considering the SMOG formula as the gold standard of measuring readability, Fitzsimmons et al(Fitzsimmons et al., 2010) interpreted the difference between the SMOG Grade and the Flesch-Kincaid Grade Level as the latter’s underestimation of a text’s readability. In their research, they were able to determine that the Flesch-Kincaid formula resulted to a mean underestimation of 2.52 grades in determining the readability of online information on Parkinson’s disease. Hence, to avoid underestimation, they suggested that the SMOG formula should be generally preferred when assessing online health information.

1.4 SMOG, Twitter, and Integrating Readability

TIME Magazine has recently created a web application for determining how smart a given tweet is by computing for its SMOG Grade (TIME, 2015a, b). Using the web application, TIME has named the top 50 smartest celebrities on Twitter by analyzing the 500 most followed twitter users’ tweets and comparing their SMOG Grades (TIME, 2015a). Similarly, 1 million tweets were analyzed using the SMOG formula and results showed that 33% of the sampled tweets are only at the fourth grade level (TIME, 2015b). TIME has argued that the 140-character limit to a tweet makes it difficult, but not impossible, for a Twitter user to compose a tweet that has a high SMOG grade. And while the findings of the analysis showed that politicians are the ones who tend to tweet using polysyllables, the results should not be treated as conclusive since the study did not follow proper sampling techniques (Steel and Torrie, 1980). Nevertheless, the potential use of SMOG in assessing the readability of tweets is highlighted in TIME’s study.

Additionally, while SMOG is tailored for longer texts, a SMOG formula for short texts has already been developed (University, n.d.). Hence, it is deemed appropriate to use SMOG in analyzing the readability of tweets which are, by nature, short texts. The SMOG formula for short texts is given in Equation 3 below:

(3)

However, up to date, the actual use of SMOG in assessing the readability of tweets has not been exhaustedly studied. Although, Guo et al(Guo et al., 2011) have already suggested integrating readability on the Twitter search engine by embedding the readability scores into the search results using the following steps:

  1. Accessing Twitter;

  2. Requesting for a Twitter archive;

  3. Parsing tweets;

  4. Computing Readability;

  5. Embedding scores into the search results.

Integrating readability in Twitter can potentially enhance the retrieval of relevant data for academic and/or commercial purposes (Guo et al., 2011), especially now that data mining has become the subject of numerous scientific studies and market research. But the reliability and effectiveness of the readability assessment must be the foremost consideration; hence, it is of utmost importance that the readability tests and tools that will be used in any assessment are context-based and yields valid and reliable results.

2 Sentiment Analysis on Twitter

Nowadays, sentiment analysis is often used as an opinion-mining tool in various social media platforms, especially in Twitter (Pak and Paroubek, 2010). Modelling the public’s mood and certain mapping socio-economic phenomena (Bollen et al., 2011) are also some of the rationale behind the plethora of sentiment analysis researches involving the high-traffic social media platform.

A number of sentiment analysis approaches had already been employed in the past. Some of which are phrase-level sentiment polarity (Wilson et al., 2005) and semantic orientation (Turney, 2002)

. Moreover, in doing sentiment analysis of tweets, a Naïve Bayes classifier is often utilized 

(Go et al., 2009; Jose et al., 2010). While the three-way classification (Go et al., 2009) has become popular over the course of time, some studies (Bollen et al., 2011) implement psychometric instruments to classify words into, not only three, but six moods or sentiments.

For the purpose of simplification and appropriating our methodology with the length of texts under study (tweets), we used a Naïve Bayes classifier for sentiment analysis.

3 Methodology

Twitter provided an Application Programming Interface (API) to allow for the automatic “scraping” of Twitter microposts (Dorsey et al., 2006). Scraping is the process of extracting pertinent data from web pages obtained from “crawling” the Internet. Crawling a set of  web pages means downloading the subset web pages given the initial web page . From the respective uniform resource locator (URL) links in hypertext markup language (HTML) anchor tags found in a web page , the next web page can be obtained and whose respective data can be scraped, . The Twitter API is a set of computer commands provided by the Twitter developers for exclusive use of programmers to allow them to tap into the Twitter data stream and gather tweets at a specific timeframe and geo-location (Dorsey et al., 2006). Once the streamed tweets have been collected by a computer program  that uses the API, they will become inputs  to two automated classifiers  and  which will respectively output the SMOG grade and contextual sentiment polarity of the tweets (Figure 1).

Figure 1: The functional relationships between the Twitter and its API, and the computer programs , , and

developed in this research to estimate the SMOG grade and sentiment of a tweet. Arrows mean the direction of data or request for data, while boxes means computer programs or commands.

Using Twitter API v1.1 in , the tweets of six Philippine senators, whose Twitter accounts were listed as verified by the Official Gazette, were collected. All tweets from August 15, 2013 to August 15, 2014 of Pia Cayetano, Miriam Defensor-Santiago, Chiz Escudero, Kiko Pangilinan, TG Guingona, and Bongbong Marcos were processed by  to become separate inputs  to  and .

Building on the PHP class called Text Statistics developed by Child (Child, n.d.), the classifier  that calculates the SMOG Grade of short texts was developed. Corrections to appropriately compute for the readability of short texts (i.e., tweets) were made on the original computer code by Child.

For the sentiment analysis , the tweets’ polarities were identified using a Naïve Bayesian classifier that classifies a given word’s sentiment as positive, negative, or neutral. Several unambiguous English and Filipino words were collated, assigned with a polarity classification, and used as library for the sentiment analysis.

4 Results and Discussion

4.1 SMOG Readability of Senatorial Tweets

All Twitter accounts of the six senators showed a high level of SMOG readability. Their respective average SMOG Grades are shown in Figure 2.

Figure 2: The average SMOG Grade of each senator over the observation period.

The lowest average SMOG Grade computed was 8.64 (Marcos). The highest, 9.22, has a small margin of difference than the rest of the computed values: 9.11, 9.15, 9.17, 9.18. This means that on the average, the senators’ tweets will be most comprehensible to those who have already completed eight to ten years of formal education. In the newly-implemented Philippine educational system (K-12), that is equivalent to late elementary school to early junior high school.

4.2 Time-dependent SMOG Readability Assessment

To find out whether the SMOG Grades of the six senators’ tweets shift through time, the computed SMOG Grades were averaged per month and are presented as Figure 3.

Figure 3: The monthly average SMOG Grade of each senator showing their respective trends over the 13-month observation period.

The SMOG Grade trend of Cayetano, Escudero, Guingona, Marcos, Pangilinan, and Defensor-Santiago varied closely. This means that the style employed by the senators in writing their posts do not vary that much and rarely shift over time. Although, Cayetano’s and Defensor-Santiago’s Twitter accounts showed a significant downward shift in readability around August-September 2013 and February-June 2014, respectively.

4.3 Sentiment Analysis of Senatorial Tweets

Results of the sentiment analysis showed that most of the senators’ tweets are neutral, otherwise positive. A breakdown of the dominant sentiment for each senator is presented in Figure 4.

Figure 4: The total number of tweets for each senator according to sentiment polarity.

Among all senators, only Cayetano tweets mostly positive messages. Marcos, on the other hand, tweets positive and neutral messages equally. Moreover, analysis of his tweets revealed that the senator virtually does not tweet negatively.

Analysis of the tweets’ sentiment vis-á-vis the senator’s gender revealed that more male senators tweet neutral tweets and that female senators tend to tweet both positively and neutrally (Figure 5).

Figure 5: The average number of tweets by gender according to sentiment polarity.

4.4 Time-dependent Sentiment Analysis

Each senator’s tweets per month were analyzed and results showed that the sentiment of some senators’ tweets vary in unison (Figure 6).

Figure 6: Monthly dominant sentiment tweets of the senators.

Cayetano, Pangilinan, Guingona, and Defensor-Santiago’s tweets went from positive to neutral around November 2013, the month of All Soul’s Day celebration, and went back to positive around December 2013 to January 2014, usually a time of celebration for Filipinos due to Chirstmas and New Year’s Day.

Moreover, the sentiments of Guingona, Pangilinan, and Defensor-Santiago’s shifted downward, although in different slopes, around February 2014 and stayed neutral until around July 2014. It is also of particular interest to note that Defensor-Santiago’s tweets around February 2014 are mostly negative.

Cayetano and Marcos’ tweets, on the other hand, shifted from neutral to positive around the month of May of 2014 and noticeably went down around the month of July, when Guingona, Defensor-Santiago, and Pangilinan’s tweet sentiments went up.

5 Conclusions

Our findings showed that Senators Marcos, Escudero, Pangilinan, Defensor-Santiago, Cayetano, and Guingona’s tweets are, on the average, between a SMOG Grade of eight to ten. Moreover, a time-dependent analysis revealed that the SMOG Grades of the senators’ tweets do not vary that much over time. This means that, the audiences who would understand a senator’s tweets are those who have attained at least the sixth grade level of education (if preparatory school is considered).

Social media users nowadays are largely composed of audience groups around that age and level of education. Hence, we deem the eight-to-ten range of SMOG Grade appropriate to the potential, if not prospective, audiences of the senators. However, ours being an exploratory study, we recommend that a more extensive research with a larger data set be done to increase the validity of such conclusion. Nevertheless, if the senators would like to expand the reach of their social media following, especially in Twitter, a SMOG Grade range of eight to ten may prove to be narrow than what would otherwise help achieve such goal.

On the other hand, sentiment analysis of the senators’ tweets revealed that most of them post neutral messages and positive, otherwise. Although five out of the six senatorial Twitter accounts that were assessed revealed a few negative sentiments. This could mean that the senators, being public figures, rarely posts negative messages as a form of cautious act. Moreover, most of these senators do not personally handle their Twitter accounts and it is their communications staff who actually post on their behalf; therefore, the neutral or positive posts could very well be considered as a digital online presence effort rather than public communication per se.

Furthermore, the fact that some of the senators’ tweet sentiments vary in unison during particular periods of time could mean that events, be it political or not, potentially affect the messages’ sentiment. For example, four senators tweeted neutral messages during November 2013 and tweeted positively from December 2013 to January 2014. Coincidentally, Filipinos are known to be very appreciative of the Christmas and New Year’s season which could be one explanation why the senators’ tweets shifted from neutral to positive during those periods. However, to be able to correlate these two variables scientifically, it is suggested that succeeding studies make use of larger data sets and a more extensive sentiment analysis tool.

6 Acknowledgements

This research effort is funded partly by and was conducted at the Research Collaboratory for Advanced Intelligent Systems, Institute of Computer Science, University of the Philippines Los Baños, College, Laguna.

References

  • Abrams (1963) M. Abrams. Education, Social Class and Newspaper Reading. Institute of Practitioners in Advertising, London, 1963.
  • Ache and Wallace (2012) K.A. Ache and L.S. Wallace. Are end-of-life patient education materials readable? Palliative Medicine, 23(6):545–548, 2012. DOI: 10.1177/0269216309106313.
  • Auta et al. (2011a) A. Auta, D. Shalkur, S.B. Banwat, and D.W. Dayom. Readability of malaria medicine information leaflets in Nigeria. Tropical Journal of Pharmaceutical Research, 10(5):631–635, 2011a. DOI: 10.4314/tjpr.v10i5.12.
  • Auta et al. (2011b) A. Auta, D. Shalkur, D.W. Dayom, and S.B. Banwat. Readability of over-the-counter medical information leaflets in Nigeria. International Journal of Pharmaceutical Frontier Research, 1(2):61–67, 2011b.
  • Bollen et al. (2011) J. Bollen, H. Mao, and A. Pepe. Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (ICWSM-11), Barcelona, Spain, 2011.
  • Brandt et al. (2005) H.M. Brandt, D.H. McCree, L.L. Lindley, P.A. Sharpe, and B.E. Hutto. An evaluation of printed HPV educational materials. Cancer Control Journal, 12(5):103–106, 2005.
  • Cherla et al. (2012) D.V. Cherla, S. Sanghvi, O.J. Choudhry, J.K. Liu, and J.A. Eloy. Readability assessment of Internet-based patient education materials related to endoscopic sinus surgery. The Laryngoscope, 122(8):1649–1654, 2012. DOI: 10.1002/lary.23424.
  • Child (n.d.) D. Child. Test statistics, n.d. https://github.com/DaveChild/Text-Statistics.
  • Coyne et al. (2003) C.A. Coyne, R. Xu, P. Raich, K. Plomer, M. Dignan, L.B. Wenzel, D. Fairclough, T. Habermann, L. Schnell, S. Quella, and D. Cella. Randomized, controlled trial of an easy-to-read informed consent statement for clinical trial participation: A study of the Eastern Cooperative Oncology Group. Journal of Clinical Oncology, 21:836–842, 2003. DOI: 10.1200/JCO.2003.07.022.
  • Dorsey et al. (2006) J. Dorsey, N. Glass, B. Stone, and E. Williams. Twitter, 2006. http://www.twitter.com.
  • Edmunds et al. (2014) M.R. Edmunds, A.K. Denniston, K. Boelaert, J.A. Franklyn, and O.M. Durrani. Patient information in Graves’ disease and thyroid-associated ophthalmopathy: Readability assessment of online resources. Thyroid, 24(1):67–72, 2014.
  • Fitzsimmons et al. (2010) P.R. Fitzsimmons, B.D. Michael, J.L. Hulley, and G.O. Scott. A readability assessment of online Parkinson’s disease information. Journal of the Royal College of Physicians of Edinburgh, 40:292–296, 2010. DOI: 10.4997/JRCPE.2010.
  • Gill et al. (2012) P.S. Gill, S. Tejkaran, A. Kamath, and B. Whisnant. Readability assessment of concussion and traumatic brain injury publications by Centers for Disease Control and Prevention. International Journal of General Medicine, 5(2):923–933, 2012.
  • Go et al. (2009) A. Go, L. Huang, and R. Bhayani. Twitter sentiment analysis, 2009. Unpublished CS224N Final Project, Stanford University.
  • Guo et al. (2011) S. Guo, G. Zhang, and R. Zhai. Integrating readability index into Twitter search engine. British Journal of Educational Technology, 42(5):E103–E105, 2011. DOI: 10.1111/ j.1467-8535.2011.01206.x.
  • Holt et al. (2010) C. Holt, T.A. Wynn, P. Southward, M.S. Litaker, J. Sanford, and E schulz. Development of a spiritually based educational intervention to increase informed decision making for prostate cancer screening among church-attending African American men. Journal of Health Communication: International Perspectives, 14(6):590–604, 2010. DOI: 10.1080/10810730903120534.
  • Hoppe (2010) I.C. Hoppe. Readability of patient information regarding breast cancer prevention from the website of the National Cancer Institute. Journal of Cancer Education, 25(4):490–492, 2010. DOI: 10.1007/s13187-010-0101-2.
  • Jose et al. (2010) A.K. Jose, N. Bhatia, and S. Krishna. Twitter sentiment analysis, 2010. Unpublished Bachelor of Technology Project, National Institute of Technology Calicut.
  • Kandula et al. (2010) S. Kandula, D. Curtis, and Q. Zeng-Treitler. A semantic and syntactic text simplification tool for health content. In Proceedings of the 2010 American Medical Informatics Association Annual Symposium, pages 366–370, 2010.
  • Leroy et al. (2013) G. Leroy, J.E. Endicott, D. Kauchak, O. Mouradi, and M. Just. User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention. Journal of Medical Internet Research, 15(7):e144, 2013. DOI: 10.2196/jmir.2569.
  • Makosky et al. (2009) D.C. Makosky, P. Cowan, N.L. Nollen, K.A. Greiner, and W.S. Choi. Assessing the scientific accuracy, readability, and cultural appropriateness of a culturally targeted smoking cessation program for American Indians. Health Promotion Practices, 10(3):386–393, 2009. DOI: 10.1177/1524839907301407.
  • McLaughlin (1969) G.H. McLaughlin. SMOG grading: A new readability formula. Journal of Reading, 12(8):639–646, 1969.
  • McLaughlin (1968) G.H. McLaughlin. Proposals for british readability measures. In J. Downing and A.L. Brown, editors, The Third International Reading Symposium, pages 186–205. Cassell, London, 1968.
  • Neuhauser et al. (2009) L. Neuhauser, B. Rothschild, C. Graham, S.L. Ivey, and S. Konishi. Participatory design of mass health communication in three languages for seniors and people with disabilities on Medicaid. American Journal of Public Health, 99(12):2188–2195, 2009. DOI: 10.2105/AJPH.2008.155648.
  • Pak and Paroubek (2010) A. Pak and P. Paroubek. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC 2010), Valletta, Malta, 2010.
  • Patel and Simpson (2010) S.R. Patel and H.B. Simpson. Patient preferences for OCD treatment. Journal of Clinical Psychiatry, 71(11):1434–1439, 2010. DOI: 10.4088/JCP.09m05537blu.
  • Steel and Torrie (1980) R.G.D. Steel and J.H. Torrie. Principles and Procedures of Statistics. McGraw Hill Text, 1980. ISBN: 007060925X.
  • Stossel et al. (2012) L.M. Stossel, N. Segar, P. Gliatto, R. Fallar, and R. Karani. Readability of patient education materials available at the point of care. Journal of General Internal Medicine, 27(9):1165–1170, 2012. DOI: 10.1007/s11606-012-2046-0.
  • Svider et al. (2013) P.F. Svider, N. Agarwal N., O.J Choudhry, A.F. Hajart, S. Baredes S, J.K. Liu, and J.A. Eloy. Readability assessment of online patient education materials from academic otolaryngology-head and neck surgery. American Journal of Otolaryngology, 34(1):31–35, 2013. DOI: 10.1016/j.amjoto.2012.08.001.
  • Swartz (2010) E.N. Swartz. The readability of paediatric patient information materials: Are families satisfied with our handouts and brochures? Paediatrics and Child Health, 15(8):509–513, 2010. PMCID: PMC2952517.
  • TIME (2015a) TIME. Smartest celebrities twitter, 2015a. Retrieved from http://time.com/2988037/smartest-celebrities-twitter/.
  • TIME (2015b) TIME. Twitter reading level, 2015b. Retrieved from http://time.com/2958650/twitter-reading-level/.
  • Turney (2002) P.D. Turney. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 417–424, 2002. DOI: 10.3115/1073083.1073153.
  • University (n.d.) Harvard University. SMOG: Assessing the readability of prose, n.d. http://www.hsph.harvard.edu.
  • Wallace et al. (2010) L.S. Wallace, A.J. Keenum, and J.E. DeVoe. Evaluation of consumer medical information and oral liquid measuring devices accompanying pediatric prescriptions. Academic Pediatrics, 10(4):224–227, 2010. DOI: 10.1016/j.acap.2010.04.001.
  • Walsh and Volsko (2008) T.M. Walsh and T.A. Volsko. Readability assessment of Internet-based consumer health information. Respiratory Care, 53(10):1310–1315, 2008.
  • Wilson (2009) M.E.G. Wilson. Readability and patient education materials used for low-income populations. Clinical Nurse Specialist, 23(1):33–40, 2009. DOI: 10.1097/01.NUR.0000343079. 50214.31.
  • Wilson et al. (2005) T. Wilson, J. Wiebe, and P. Hoffmann. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347–354, 2005. DOI: 10.3115/ 1220575.1220619.