Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics

12/14/2018 ∙ by Lisa Posch, et al. ∙ 0

Micro-task crowdsourcing is an international phenomenon that has emerged during the past decade. This paper sets out to explore the characteristics of the international crowd workforce and provides a cross-national comparison of the crowd workforce in ten countries. We provide an analysis and comparison of demographic characteristics and shed light on the significance of micro-task income for workers in different countries. This study is the first large-scale country-level analysis of the characteristics of workers on the platform Figure Eight (formerly CrowdFlower), one of the two platforms dominating the micro-task market. We find large differences between the characteristics of the crowd workforces of different countries, both regarding demography and regarding the importance of micro-task income for workers. Furthermore, we find that the composition of the workforce in the ten countries was largely stable across samples taken at different points in time.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 11

page 14

page 17

page 18

page 20

page 22

page 23

page 25

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Doing freelance work over an online platform or by other digital means is a mode of labor that has existed since at least the 1990s. However, doing such work (i) for many, largely unknown principals, (ii) on very small-scale, non-expert tasks (often in parallel with other workers) that (iii) are continuously available from central platforms is a development that has only gained increasing popularity with an international workforce over roughly the past 15 years and has no offline equivalent. This mode of work is called micro-task crowdsourcing and, through its low-barrier nature, offers potential income opportunities for almost everyone with an internet connection.111Other modes of crowdsourcing exist, such as winner-takes-all contests for ideas or more macro-size tasks, essentially constituting online freelancing (cf. Kuek et al. (2015)).

Precise estimates of the number of platforms, their users and turnovers for this type of work are hard to come by, as no official labor market statistics for crowdwork exist as of yet, and proprietary platforms seldom release such information. However, experts postulate a significant and lasting growth of microtask platforms, assuming a market size of $500 million in 2016, with the amount of global micro-task workers being put at around 9 million, up from 4 million in 2013

Kuek et al. (2015). The World Bank Kuek et al. (2015), the European Agency for Health and Safety at Work European Agency for Health and Safety at Work (2015) and other official bodies have in recent years been discussing chances and perils of this new form of income for millions of people, and they see the need for better regulation, but also plainly for better insights into the crowdsourcing market. Scholars and legislators have for instance expressed qualms about the tendency of crowdwork – often meant to offer supplementary income – to evolve into a main income source for workers in precarious economic circumstances, while at the same time being unregulated, volatile in terms of pay and availability, not offering union-typical bargaining powers and requiring predominantly monotonous work. On the flip side, opportunities through crowdwork have been highlighted, especially for inhabitants of regions with sub-par working conditions in “offline” employment Kuek et al. (2015).

To inform this discussion of the impact of crowdwork on communities around the world, research concerned with the demographic composition of the international crowd workforce is very valuable, not least to enable comparisons with the more traditional, offline workforce. In this regard it is also strongly linked with the study of why crowdworkers are attracted to this new form of employment (e.g. Posch et al. (2017); Brewer et al. (2016); Brabham (2010)).

Further, demographic information is instrumental for optimizing the use of crowd platforms as recruiting instruments to infer knowledge about a broader ground population, or at least control for sampling biases – e.g., for using crowdworkers as an affordable and expeditious alternative in psychological testing Paolacci and Chandler (2014). Lastly, it is valuable for understanding task performance linked to demographic features, e.g., for labeling, translation, or speech recognition tasks (e.g. Kazai et al. (2012); Pavlick et al. (2014)).

While its useful applications seem apparent, knowledge about the demographic composition of the crowd workforce remains spotty. Out of the two mayor micro-task platforms dominating the market,222MTurk and Figure Eight (CrowdFlower) are estimated to share 80% of all revenue generated in the microtask market, with revenues approximately equal Kuek et al. (2015). only the demographic composition of the predominantly American and Indian crowdworkers on Amazon Mechanical Turk (MTurk) is sufficiently well-known (e.g. Ipeirotis (2010b); Ross et al. (2010); Berg (2016)), but insights about other platforms – and particularly workers in countries outside the MTurk target audiences – are few and far between.

This paper therefore sets out to complement the existing literature by mapping out the demographics of the second micro-task market leader, CrowdFlower (since 2018 known as Figure Eight),333 The platform’s name changed from CrowdFlower to Figure Eight in 2018 and at the time of our data collection, which started in 2016, the platform’s name was CrowdFlower. For consistency with the survey questions, we therefore refer to the platform as CrowdFlower rather than Figure Eight in the remainder of this paper. exploring its much more international crowd workforce to shed additional light on country-specific differences. We conducted a survey of CrowdFlower workers in ten countries, over two time points, collecting information about their demography as well as the centrality of micro tasks in their life, regarding time spent as well as importance and use of micro-task income.

The main contributions of this paper are (1) a large-scale comparison of crowdworker demographics in ten different countries, (2) a comparison of the centrality of micro tasks in the worker’s lives in these ten countries and (3) an analysis of the changes in these features between two samples taken eight months apart.

The paper is structured in the following way. Section 2 gives an overview of related work on the characteristics of crowdworkers. Section 3 describes our survey design and the process of data collection. In Section 4, we present a cross-national comparison of crowdworker demographics, and Section 5 presents a comparison of the importance that micro tasks have for workers in different countries. Finally, Section 6 concludes this paper.

2 Related Work

Most research investigating demographic and economic characteristics of workers on micro-task platforms has focused on the platform Amazon Mechanical Turk (MTurk). Early studies on the demographics of workers on MTurk Ipeirotis (2010b); Ross et al. (2009); Ross et al. (2010); Paolacci et al. (2010); Kazai et al. (2012) found that the vast majority of workers were located in the USA and India, and that they were young and highly educated. Workers were predominantly female in the USA and predominantly male in India. A small but significant percentage of workers relied on MTurk to make basic ends meet.

Later studies on the demographics of MTurk workers reported similar results (e.g. Goodman and Paolacci (2017); Berg (2016); Peer et al. (2017); Pavlick et al. (2014); Naderi (2018); Difallah et al. (2018)). On MTurk, American and Indian crowdworkers still constitute the vast majority of workers,444Crowdworkers from the United States and from India constitute over 80% of the worker population on ATM (also see http://demographics.mturk-tracker.com/##/countries/all). which is likely due to the fact that workers from other countries can only receive payment from MTurk in the form of Amazon.com gift cards Amazon Mechanical Turk (2016). Consistent with earlier studies, Berg (2016) found that Indian and American workers on MTurk were young and well-educated. Indian workers were predominantly male, but there was now more gender balance among workers from the US. These findings are also supported by current data collected by mturk tracker555http://www.mturk-tracker.com Ipeirotis (2010a). Pavlick et al. (2014) conducted a study on the languages spoken by bilingual workers on MTurk and found that the majority of workers who accepted their translation tasks were located in either the USA or India. Nevertheless, there were sufficient bilingual workers to accurately and quickly complete translation tasks for 13 different languages.

Research on the demographics of workers on MTurk is closely linked with questions concerning the representativeness of MTurk samples and their suitability for different research purposes (e.g. Goodman and Paolacci (2017)). For example, Paolacci et al. (2010) compared American crowdworkers on MTurk to the general US population and found that workers in the USA were more representative of the population than university subject pools. Compared to the general US population, crowdworkers tended to be slightly younger and, despite being more highly educated, workers had a lower income level. This observation could be partially explained by age. Buhrmester et al. (2011) compared MTurk workers to standard Internet samples. Their MTurk sample was more diverse than both standard Internet samples and American college samples. They found that MTurk workers were similar in gender distribution, more non-white, almost equally non-American, and older than the standard Internet sample. Berinsky et al. (2012)

evaluated the suitability of crowdworker samples for experimental political science and found that the respondents recruited on MTurk were more representative of the U.S. population than in-person convenience samples, but less representative than respondents recruited for Internet-based panels or national probability samples. Furthermore, they found that crowdworkers responded to experimental stimuli in a way that was consistent with prior research.

Weinberg et al. (2014) analyzed sociodemographic characteristics of workers on MTurk and compared them to the characteristics of respondents of a population-based web panel. They found that the MTurk participants were younger, more educated and there was a higher proportion of women than among the web panel participants. The MTurk sample was more divergent from the general population than the web panel. Huff and Tingley (2015) analyzed the demographics and political characteristics of MTurk workers from the United States and compared them to the respondents of the Cooperative Congressional Election Survey (CCES), a stratified sample survey conducted yearly in the United States. They found that MTurk was, in many cases, good at attracting those demographics that were difficult to attract for CCES (e.g. young Asian males). Furthermore, they found that the distribution of employment in different occupational sectors of workers on MTurk was very similar to that of CCES respondents, and that the respondents were located in similar locations on the rural-urban continuum.

Shapiro et al. (2013) investigated the suitability of crowdworker samples for conducting research on psychopathology, investigating the prevalence of different psychiatric disorders and related problems among crowdworkers on MTurk. They concluded that MTurk might be useful for studying clinical and subclinical populations. Paolacci and Chandler (2014) analyzed the characteristics of MTurk as a participant pool for the social sciences and concluded that worker samples from MTurk could replace or supplement convenience samples in psychological research, but that they should not be considered representative of a country’s population.

Research on the demographics of workers on other micro-task platforms, and therefore also on workers based in countries other than the USA and India, is more scarce. Furthermore, due to reasons such as unavailability of demographic data beyond the workers’ location or small sample sizes, none of these studies have so far analyzed and compared the demographics of workers at the country level.

Hirth et al. (2011) analyzed the demographics of the platform Microworkers with respect to the home countries of requesters and workers and found that the platform was much more geographically diverse than MTurk. The countries with the largest amount of workers were Indonesia, Bangladesh, India and the United States, accounting for 60% of the workforce on the platform. The remaining 40% were dispersed over a heterogeneous set of geographical locations. Using the United Nations Human Development Index to categorize countries, they found that an almost equal proportion of workers were located in countries with low development (24%) and countries with very high development (21%), while the majority of the workforce was located in countries with medium development (45%).

Martin et al. (2017) compared the demographics of workers on MTurk to the demographics of workers on the platforms Microworkers and Crowdee. The study grouped workers on the Microworkers platform into workers located in “Western countries” (including all workers from Europe, Oceania and North America) and workers located in “developing countries” (including all workers from South America, Asia and Africa). The results of their demographic survey indicated that workers on Microworkers and Crowdee were predominantly male, younger than workers on MTurk and highly educated. A large proportion of workers reported working either full-time or part time on all three platforms. Regarding the differences between “Western countries” and “developing countries,” the study found that workers in the “developing countries” group were younger, lived in larger households, were more educated, had a lower household income and spent more time on the platform, compared to workers in the “Western countries” group.

Further, few studies have concerned themselves with the workforce demographics of the platform CrowdFlower, which covers half the market share for micro-tasks and traditionally employs a geographically diverse set of workers.

Berg (2016) collected demographic data from a geographically diverse sample of workers on CrowdFlower and found that only 2.8% of workers (10 respondents) were from the US and 8.5% were from India (30 respondents). The workers on CrowdFlower were predominantly male and more educated than American workers on MTurk, but less educated than Indian workers on MTurk. Peer et al. (2017) examined the demographics of workers on CrowdFlower and Prolific Academic and compared them to the demographics of workers on MTurk. The study found that, compared to MTurk, both CrowdFlower and Prolific Academic had a higher proportion of male workers and the mean age was similar on all three platforms. CrowdFlower had the highest diversity in terms of race and both CrowdFlower and Prolific Academic were much more diverse in worker location than MTurk. On all three platforms, workers were highly educated.

In sum, there have been extensive studies on the characteristics and demographic composition of American and Indian workers on MTurk, whereas research on the characteristics of crowdworkers on other micro-task platforms and on the demographic composition and characteristics of workers in countries other than the USA and India remains sparse. In comparison, we provide a survey of workers pre-selected to cover similar respondent numbers for ten diverse countries, over two time points, whereas previous studies have studied samples that were not stratified by locations, and have mostly not controlled for temporal changes. In doing so, we have conducted the most comprehensive scientific collection of worker characteristics on CrowdFlower to date, with 11,946 individual responses collected (after spam removal). Using the data collected with our survey, we provide the first country-level comparison of (i) the demographics and (ii) the centrality of micro tasks in the lives of crowdworkers on this platform and show that notable differences exist between countries.

3 Survey

In order to provide insights into the characteristics of the international crowd workforce, we conducted a large survey in ten different countries and at two points in time on the CrowdFlower platform.

Group Code
All ALL 18000 35 % 5857 32% 6089
High Income HIGH 5400 28 % 1952 26% 1988
Middle Income MID 5400 32 % 1834 31% 1863
Low Income LOW 5400 44 % 1508 38% 1679
USA USA 1800 20 % 721 14% 776
Spain ESP 1800 25 % 677 30% 634
Germany DEU 1800 38 % 554 36% 578
Brazil BRA 1800 45 % 496 43% 509
Russia RUS 1800 25 % 677 21% 708
Mexico MEX 1800 27 % 661 28% 646
India IND 1800 32 % 608 28% 645
Indonesia IDN 1800 55 % 401 47% 476
Philippines PHL 1800 45 % 499 38% 558
Venezuela VEN 1800 37 % 563 38% 559
Table 1: Sample sizes and percentage of spam received. This table shows the sample sizes of the different groups at both time points, as well as the percentage of spam received. shows the total number of responses collected, before removing spam. and show the number of responses after spam removal in sample T1 and T2, respectively. and show the percentage of workers who did not pass all attention checks, for T1 and T2.

3.1 Data Collection

We posted the survey as a micro task on CrowdFlower. The task included seven questions about the workers’ demographics and three questions about the centrality of micro tasks in the workers’ lives. Furthermore, the task contained questions about the workers’ motivation for putting effort into micro tasks, which were used for the validation of the Multidimensional Crowdworker Motivation Scale Posch et al. (2017). Anonymity was ensured in the task instructions.

We collected data of workers from ten countries, with 900 participants in each country at each time point. In our country selection, we aimed for diverse income levels by selecting countries from three different World Bank income groups.666The World Bank country classification is available at http://databank.worldbank.org/data/download/site-content/CLASS.xls. Here, we use the group label “middle Income” (MID) for the upper middle income group and “low Income” (LOW) for the lower middle income group for better readability. Furthermore, we aimed for a high cultural diversity and sufficient activity on CrowdFlower.777The country had to either be high in the Alexa (http://www.alexa.com/) ranking or one of the top contributing countries in at least one of CrowdFlower’s partner channels. The countries that we selected for the high income group were USA, Germany and Spain, for the middle income group we selected Brazil, Russia and Mexico, and for the low income group we selected India, Indonesia and the Philippines. Additionally, we collected data from Venezuela because it was the most active country on CrowdFlower at the time of the start of the data collection. However, Venezuela represents a special case concerning income due to the circumstance that the black market exchange rate deviates from the official exchange rate to a large extent Bloomberg News (2016). Therefore, we did not include Venezuela in any of our income groups.

We posted the survey on CrowdFlower at different times during the day and the week, in order to capture a diverse sample of workers.888There are indications that worker composition varies by time of the day and day of the week, see e.g. http://demographics.mturk-tracker.com For each country, 300 responses were requested during typical working hours (8:00 am to 5:00 pm in the appropriate time zone), 300 responses were requested in the evening (6:00 pm to 11:00 pm in the appropriate time zone), and 300 responses were requested during weekends. After the first round of data collection, which took place in October and November 2016 (T1), we conducted a second round of data collection in June and July 2017 (T2). The survey was conducted in English in all countries. While this approach only captures crowdworkers with sufficient English skills, demand for crowdworkers is driven by Anglophone clients and English is the dominant language in task requests Kuek et al. (2015). Congruently, English is expected by CrowdFlower to be spoken by all workers at a sufficient level to solve tasks, as made apparent by its interface language and English being assumed a guaranteed language skill for all workers in the platform’s worker language selection settings.

Demographics
D1 What is your gender?
D2 What is your age?
D3 What is your marital status?
D4 How many people live in your household?
D5 What is your highest education level?
D6 What is your employment status (CrowdFlower tasks excluded)?
D7 What is your approximate household income, per year (after taxes, in US$)?
Importance of Micro Tasks
I1 How much time do you spend on CrowdFlower, per week?
I2 Is the money from CrowdFlower your primary source of income?
I3 What do you do with the money that you earn on CrowdFlower?
Table 2: Survey Questions. This table shows the survey questions in our CrowdFlower task.
Basic Expenses
“I buy food!!” (USA), “I use the money to help pay my monthly rent.” (USA), “buy sensors to glucose measure” (Spain), “I use it for my daily needs like to pay rent and buy my essentials.” (India), “use it for my medicines” (India)
Leisure Activities
“I will use to pay for my hobbies.” (USA), “entertainment, eating out” (USA), “Go to the cinema.” (Spain), “With the money I usually do trips.” (Spain), “Use it as pocket money” (India)
Save/Invest
“Put it in a savings account” (USA), “Build a BitCoin investment portfolio” (USA), “I keep it for the future” (Spain)”, “I save all the money I earn” (Spain), “i save the money for investments” (India), “saving for marriage and future life” (India)
Buy Gifts
“I will save it to try and afford a gift for my children for christmas” (USA), “spend it on xmas presents for my kid” (USA), “small Gifts” (Spain), “buy gifts to my four daughters” (Spain), “Use it to buy gifts for my children.” (India), “used it for my mom dad’s anniversary” (India)
Education
“Pay for my college tuition.” (USA), “Use it for my driving test.” (Spain), “Save it [t]o pay for my college expenses” (Spain), “The Money I have Earned in CrowdFlower is Used for my Studies.” (India), “for further studies” (India)
Donate to Charity
“I will spend for my family and remaining to charity.” (India), “I want to do lot of the things, primary is to donate a share out of it […].” (India), “Almost 80% paid for poor children’s fee.” (India), “helping to poor peoples” (India)
Other
“Nothing yet, this is my first task.” (USA), “Multiple things, nothing in particular.” (Spain), “it is very small amount to spend i have not earned so much” (India), “I have not much enough to withdraw it.” (India)
Table 3: Exemplary responses to the open ended survey question “What do you do with the money that you earn on CrowdFlower?”.

In the tasks, we included four attention checks for detecting spam, such as workers clicking randomly or accepting the task despite having insufficient English skills.999For a detailed description of the spam filtering process, see Posch et al. (2017). Table 1 shows the number of respondents per income group and country for each time point, as well as the percentage of spam received and the number of respondents after spam removal. As it is the crowdworkers’ choice whether to accept a task or not, our samples are necessarily self-selected, as is generally the case for surveys on micro-task platforms.

In the survey, we asked crowdworkers about seven demographic characteristics and about three aspects concerning the importance of micro tasks in their life. Table 2 shows the questions. The question about crowdworkers’ use of the money earned through micro tasks (I3) was constructed as a multiple choice question. We aimed for a high-level distinction of money use to keep the number of answer options low (and reduce the total survey length). As standard survey instruments for capturing expenditures of households or individuals are very detailed in their classifications and do not provide canonical distinctions at a sufficiently high level for our purposes101010Cf., e.g., the “Consumer Expenditure Survey Interview Questionnaire” U.S. Bureau of Labor Statistics (2018) or the "Classification of Individual Consumption according to Purpose" United Nations Department Of Economic And Social Affairs - Statistics Division (2000). If coarser distinctions are defined, they are generally at the binary consumption vs. savings/other expenditures level (cf. Destatis (2013)). – and to account for potential particularities of crowdworker money use patterns – we opted for an inductive approach to constructing the answer options.

To this end, we posted a preceding open ended survey task, where we asked workers the question “What do you do with the money that you earn on CrowdFlower?” For answering the question, we provided workers with a free text field for their answer, which could be arbitrarily long. We posed this question to workers in the USA, Spain and India.111111For the development of the answer options, we used the same countries as for the development of the Multidimensional Crowdworker Motivation Scale Posch et al. (2017). USA and India were selected because these countries have significant populations of crowdworkers on different platforms. Spain was selected in order to include a European country with a sufficiently large population of crowdworkers. In each country, 300 workers were surveyed in October 2016. Two authors of this paper then manually categorized the open-ended responses. Workers often reported more than one use for the money earned through micro tasks, so each answer could be coded with multiple categories.

We then used these manually identified categories to construct the answer options for the survey question I3: (1) I use the money for basic living expenses (food, rent, sanitary items, medical care,…), (2) I spend the money on leisure activities (hobbies, games, holidays, sports,…), (3) I save/invest the money, (4) I use the money to buy gifts for other people, (5) I use the money to finance my education, (6) I donate the money to charity, and (7) Other purposes. Table 3 shows example answers for each category, along with the country the answers stem from.

4 Demographics

In this section, we report the results of the demographics section of the survey (see questions D1-D7 in Table 2). For each demographic characteristic, we report the proportion of each answer choice in the ten countries as an average of T1 and T2, the differences in proportion between the countries and, per country, the differences in proportions between the two samples taken eight months apart. As a measure of difference, we report the Jensen-Shannon (JS) divergence Lin (1991) between the respective answer distributions for each demographic characteristic. The JS divergence quantifies how dissimilar two distributions are and is bounded by and . A value of indicates equivalence between the distributions and higher values indicate the degree of dissimilarity. The reported JS divergences between two countries are the averages of the divergences between these countries at T1 and T2.

4.1 Gender

Figure 1: Gender Distribution. This figure shows the gender distribution of workers in the different countries. The bar height represents the average of T1 and T2.

In most countries, crowdworkers were predominantly male, with the proportion of male workers exceeding 60%. The gender distribution was similar in all countries with the exceptions of the USA and the Philippines, which were the only two countries where female workers constituted the majority. The most gender balanced workforce was present in the Philippines, with 52% (in T1) and 55% (in T2) percent of workers being women. Figure 1 shows the gender distribution in the ten countries. The height of the bars corresponds to the average of the proportions at T1 and T2.

Figure 2: Gender JS. This figure shows the JS divergences between the gender distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

The answer options to the gender question included a third category, “other,” which is not included in Figure 1 due to the small number of responses. The differences between the sums of the male and female percentages and 100% are due to this third category.

The gender distributions of American and Indian workers are consistent with findings of early studies on MTurk (e.g. Ipeirotis (2010b); Paolacci et al. (2010)) which found that in the United States, there were more female than male workers, and in India, there were more male workers. However, the United States crowd workforce on MTurk, at the time of data collection, was more gender balanced than the US-based crowd workforce on CrowdFlower.121212For data on the gender distribution of American and Indian workers on MTurk, see http://demographics.mturk-tracker.com/##/gender/all. Ipeirotis (2010b) hypothesized that this gender distribution difference between India and the United States may be due to the fact that in the United States, MTurk is often used by stay-at-home parents and underemployed or unemployed workers (which are more likely to be female), while in India, workers are more likely to rely on MTurk as a primary source of income. However, our results show that this does not generally hold true for the differences between the gender distribution of high income and low income countries.

Figure 2 shows the JS divergences of the answer distributions between each country pair and, for each country, the divergence between T1 and T2. The gender distribution was mostly stable between the time points, with Spain and the USA exhibiting the largest differences in distributions. The divergence in the USA was mainly due to the gender category “other,” as which ten crowdworkers identified in T2, compared to none in T1. The divergence in Spain was due to an increased proportion of female workers in T2, where 35% of workers reported being female in T2 compared to 30% in T1. While the change was less pronounced in other countries, the percentage of female crowdworkers slightly increased from T1 to T2 in all countries except Russia.

4.2 Age

Figure 3: Age Distribution. This figure shows the age distribution of workers in the different countries. The percentages represent the average of T1 and T2.
Figure 4: Age JS. This figure shows the JS divergences between the age distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

Crowdworkers were young in all countries, with most crowdworkers being between 18 and 34 years of age. This is consistent with studies on MTurk (e.g. Ipeirotis (2010b); Ross et al. (2010); Berg (2016)), which found that younger workers were overrepresented on the platform.

The country with the oldest population of crowdworkers on CrowdFlower was Russia, which had by far the lowest proportion of workers aged between 18-24 years and the highest population of workers aged between 35 and 54 years old. Venezuela had the highest proportion of very young workers (aged 18-24). Figure 3 shows the age distribution in the ten countries. Data from mturk tracker131313http://demographics.mturk-tracker.com/##/yearOfBirth/all indicates that Indian workers on MTurk tend to be younger than American workers. This difference also seems to be present on CrowdFlower, especially for the proportion of workers aged 18 to 24 years, which formed a much higher percentage of the Indian crowd workforce than the American crowd workforce.

Figure 4 shows the JS divergences of the age distributions. In most countries, there was little difference in age distribution between T1 and T2. Venezuela had the largest difference in age distribution between the two time points, mostly due to an increase in young workers in the age bracket 18 to 24 years and a decrease in workers aged over 24.

4.3 Marital Status

Most countries had a higher proportion of non-married workers than married workers. Of all countries, Russia had the highest proportion of married crowdworkers and it was the only country with more than 60% married workers. USA and Russia were the countries with the largest proportion of divorced or separated workers. The countries with the highest proportion of non-married workers were Germany, the Philippines and Venezuela. Figure 5 shows distribution of the workers’ marital status in the ten countries. The response option for this survey question also included the category “widowed,” which received a very small number of responses and is therefore not included in Figure 5.

Figure 6 shows the JS divergences of the answer distributions. Russia had the highest divergences with other countries due to the high proportion of married workers. Regarding the differences between the time points, in most countries there was a slight decrease in the proportion of married workers from T1 to T2. The only country where this was not the case was Germany, which also had the most stable distribution of marital status between the time points.

Figure 5: Distribution of Marital Status. This figure shows the marital status distribution of workers in the different countries. The bar height represents the average of T1 and T2.
Figure 6: Marital Status JS. This figure shows the JS divergences between the marital status distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

4.4 Household Size

Germany had the highest proportion of single and two-person households, followed by the USA. All other countries had a very low proportion of single households (below 10%). The Philippines was the country with by far the highest proportion of households with more than seven persons, with more than double the proportion of all other countries. Spain had the highest proportion of four-people households, and Russia had the highest proportion of three-person households. India’s crowd workforce reported the lowest proportion of single households. Figure 7 shows the distribution of household size.

Data from mturk tracker141414http://demographics.mturk-tracker.com/##/householdSize/all shows that on MTurk, workers in India tend to live in larger households than American workers. This difference in household size was also present in the workers on CrowdFlower. Workers located in the United States mainly lived in households with two or three persons, while a four-person household was the most common response among workers in India.

Figure 7: Distribution of Household Size. This figure shows the household size of workers in the different countries. The percentages represent the average of T1 and T2.
Figure 8: Household Size JS. This figure shows the JS divergences between the household size distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

Figure 8 shows the JS divergences of the household size distributions. Generally, we found the largest divergences between the countries of the high income group as well as Russia and the low income countries. There were no large differences in household size distribution between the two time points. The German sample had the largest difference, with more workers reporting living in two-person households in T2 than in T1, and less workers reporting three-person households.

4.5 Employment Status

The question regarding workers’ employment status asked crowdworkers to explicitly exclude their activity on CrowdFlower. Figure 9 shows the distribution of employment status.

In almost all countries, over 35% workers had a full-time job besides their activity on CrowdFlower. The only exception to this was Venezuela, where only 28% had full-time jobs at T1 besides CrowdFlower. This percentage was even lower in T2, where only 23% of Venezuelan workers reported having full-time jobs. A significant proportion of workers reported being in education, with Germany and Venezuela having the highest proportion of workers in education. The highest proportion of unemployed workers was reported in the United States, followed by Venezuela. Very few workers reported being retired, which is very likely due to the overall young age of the workers.

Figure 10 shows the JS divergences between the employment status distributions. The largest difference in employment status distribution was between Russia and Venezuela. While Russian workers reported the highest percentage of workers in full-time employment, Venezuela reported the lowest percentage of all countries. Furthermore, there were large differences between Russia and Venezuela in the proportion of workers who reported being unemployed or in education.

In most countries, less workers reported working full-time in T2 than in T1, while the percentage of unemployed workers, workers in education and part-time workers increased from T1 to T2. In Brazil, which had the largest JS divergence between the time points, this change was most pronounced, with a large decrease of workers in full-time employment (from 59.5% in T1 to 46.2% in T2) and a large increase of unemployed workers (from 13.1% in T1 to 21.6% in T2). An exception to this pattern was Germany, where the percentage of workers employed full-time stayed roughly the same, while there was a slight decrease in unemployed workers and a slight increase of workers in education. The second-largest JS divergence between time points was in Indonesia, where a lower proportion of workers reported having a full-time job in T2 than in T1, and a higher proportion of workers reported holding a part-time job.

Figure 9: Employment Status Distribution. This figure shows the employment status distribution of workers in the different countries, CrowdFlower tasks excluded. The bar height represents the average of T1 and T2.
Figure 10: Employment Status JS. This figure shows the JS divergences between the employment status distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

4.6 Education Level

Figure 11: Education Level Distribution. This figure shows the education level distribution of workers in the different countries. The bar height represents the average of T1 and T2.
Figure 12: Education Level JS. This figure shows the JS divergences between the education level distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

Crowdworkers on CrowdFlower are generally well educated. The proportion of workers having a Bachelor’s degree or higher was 30% or above in all countries. Figure 11 shows the distribution of education level.

Workers in the low income group countries reported especially high education levels. The countries with the highest proportion of college graduates were India and the Philippines. India also had the highest proportion of workers with a Master’s degree of all countries.

Our finding that crowdworkers are generally highly educated is consistent with the findings of studies on the demographics of MTurk (e.g. Berg (2016)), and it contrasts with the notion that micro-work is especially attractive to unemployed people with no specialized skills (e.g. Kuek et al. (2015)). The fact that workers from lower income countries tend to have higher education levels is consistent with the findings of studies on MTurk (e.g. Ipeirotis (2010b)), which found that Indian workers on MTurk tend to have more education than workers from the United States. An exception to this pattern seems to be Venezuela, where workers tend to be less educated than in other low income countries.151515While we did not include Venezuela in the low income country group due to the reasons stated in Section 3, Venezuelan workers reported a very low household income.

Very few workers reported having no schooling completed at all (below 2% in all countries) and only a small proportion of workers reported having only “some high school.” Germany had the highest proportion of workers with a high school degree but no college education.

Figure 12 shows the JS divergences between the education level distributions. The largest difference in distribution was between Venezuela and India, with the proportion of Indian workers with a Bachelor’s degree being more than twice as high and the proportion of workers with a Master’s degree being over three times higher than the proportion in Venezuela.

Regarding the difference between the two time points, we found the largest differences in Russia, Brazil and Venezuela. Russia had less workers with a Bachelor’s or Master’s degree in T2 than in T1. In Brazil, the proportion of workers reporting a high school degree but no college degree increased and the proportion of workers reporting some high school, a Bachelor’s degree or associate degree decreased. In Venezuela, the proportion of high school graduates with no college increased, while the proportion of workers reporting vocational training or an associate degree decreased.

4.7 Yearly Household Income

Figure 13: Distribution of Household Income. This figure shows the household income distribution of workers in the different countries. The percentages represent the average of T1 and T2.
Figure 14: Yearly Household Income JS. This figure shows the JS divergences between the household income distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

In order to meaningfully capture household income in a set of countries with wildly varying average incomes, we created logarithmic bins161616We rounded the logarithmically spaced numbers for better readability in the answer options. for the response options. The question asked workers to report an estimate of their annual disposable household income (i.e. after taxes) in US dollars. Figure 13 shows the household income distribution for each country.

Workers from Venezuela, while classified as an “upper middle income” country by the World Bank, reported by far the lowest annual household income. This supports our decision to not include Venezuela in any of the income groups. Apart from Venezuela, the reported income distributions are largely consistent with the World Bank classification of the countries, with the United States, Spain and Germany reporting higher incomes (despite the smaller reported household size) and India, Indonesia and the Philippines reporting lower incomes. Unsurprisingly, data from mturk tracker

171717http://demographics.mturk-tracker.com/##/householdIncome/all shows that on MTurk, Indian workers also tend to report lower household incomes then workers from the United States.

While the proportion of workers reporting an annual income below US$ 3,000 was much higher in low income countries than in high income countries, a significant proportion of workers in high income countries also reported a yearly house income of less than US$ 3,000. There might be several explanations for this, such as students living on student loans, unemployed workers living off their savings, or workers on welfare benefits who do not consider the benefits as “income.”

Figure 14 shows the differences between the household income distributions. The largest differences in household income were generally found between the countries in the low income group (and Venezuela) and the countries in the high income group, with the largest difference being between the USA and Venezuela.

Between T1 and T2, the household income distributions remained largely stable. We observed the largest change in Indonesia, where in T2 more workers reported a yearly income below US$ 3,000 (39%) than in T1 (30%). The second largest change between the time points was in Mexico, where the number of workers reporting a household income between US$32,000 and US$50,000 decreased from 12% to 7% while the proportion of workers reporting a lower income increased.

5 Importance of Micro Tasks for Crowdworkers

In this section, we compare the importance of micro tasks and micro-task income for workers in the ten different countries. Our survey included three questions about different aspects concerning the centrality of micro tasks in the workers’ lives (see questions I1-I3 in Table 2). Analogously to the previous section, we report the proportion of each answer choice in the ten countries as an average between T1 and T2 as well as the JS divergences Lin (1991) of the answer distributions between the countries and between the two time points.

5.1 Weekly Time Spent on CrowdFlower

Figure 15 shows, for the ten countries, how much time workers report spending on CrowdFlower per week. Venezuela, the Philippines and Indonesia were the countries with the highest proportion of workers who reported spending more than 20 hours per week on CrowdFlower and Venezuela had the highest proportion of workers spending more then 40 hours per week on the platform. In all countries, but especially in the countries in the high and middle income groups, there was a significant proportion of workers who used CrowdFlower less than two hours per week. The countries in the high income group had the highest proportion of workers who reported spending less than one hour per week on CrowdFlower.

Figure 16 shows the JS divergences between the answer distributions. Regarding the differences between countries, countries in the high income group were generally most dissimilar to countries in the low income group, with countries in the low income group generally spending more time on the platform.

Figure 15: Time Spent on CrowdFlower per Week. This figure shows the distribution of weekly time spent on the platform by workers in the different countries. The percentages represent the average of T1 and T2.
Figure 16: JS of Weekly Time Spent on CrowdFlower. This figure shows the JS divergences between the answer distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

The largest change in distribution between T1 and T2 was in Venezuela. In T2, the proportion of Venezuelan workers spending over 40 hours per week on CrowdFlower (19.5%) was almost double the proportion reported in T1 (9.6%). This increase was likely due to changes in the economic situation of the country, making CrowdFlower an increasingly attractive source of income for Venezuelan workers.

5.2 Dependency on Micro-Task Income

Figure 17: Dependency on CrowdFlower Income. This figure shows the proportion of workers who reported micro-task income being their primary/non-primary source of income in the different countries. The bar heights represent the averages of T1 and T2.
Figure 18: Dependency on CrowdFlower Income JS. This figure shows the JS divergences between the answer distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

Crowdworkers in countries of the high and middle income groups reported the lowest percentages of reliance on CrowdFlower as their primary source of income. There were no large differences between the countries of the high income group and those of the middle income group, and the lowest reliance on CrowdFlower as a main source of income was in Russia, a country in the middle income group. In the low income group as well as in Venezuela, the proportions were significantly higher. Figure 17 shows the answer distribution of each country.

In terms of distribution differences between countries, Venezuela had the highest JS divergences with other countries, especially with the countries in the high and middle income group. The countries in the high and middle income categories were very similar among each other. Figure 18 shows the JS divergences of the answer distributions.

The reliance of workers on CrowdFlower as a main source of income was mostly stable between T1 and T2, with the exception of Venezuela and, to a lesser extent, Brazil. In Venezuela, consistent with the increase of weekly time spent on the platform, the percentage of workers relying on CrowdFlower as a primary source of income significantly increased from T1 (29%) to T2 (41.5%). In Brazil the percentage was also higher in T2 (15.9%) than in T1 (11.1%).

5.3 Use of Micro-Task Income

The question regarding workers’ use of the income earned through micro tasks offered seven answer options (see Table 3) and workers could select one or more of the options. Figure 19 shows the proportion of workers who selected the different expenditure categories, for each country.181818Note that the sum of the different categories may be higher than 100% for each country, as workers could choose more than one expenditure category.

In seven out of ten countries, the proportion of workers who reported spending micro-task income on basic expenses such as food, rent, sanitary items or medical care exceeded 40%. The countries with the highest proportion of workers who spent the money on basic expenses were the Philippines and Venezuela. Germany was the country with the lowest percentage of workers spending the money for basic expenses, followed by Spain and Russia. In the USA, despite being a high income country, over 40% of workers reported spending the money on basic expenses.

The three countries in the high income group and Brazil had the highest percentage of workers who stated spending the money on leisure activities such as hobbies, games, holidays or sports. In all other countries except Venezuela, the proportion of workers who reported spending micro-task income on leisure activities was also higher than 30%. In Venezuela, the proportion of workers who reported spending micro-task income on leisure activities was by far lowest of all countries.

Figure 19: Use of CrowdFlower Income. This figure shows how workers spend their income from micro tasks in the different countries. The bar heights represent the averages of T1 and T2.
Figure 20: Use of CrowdFlower income JS. This figure shows the JS divergences between the answer distributions of the different countries. The bar on the right shows the JS divergence between T1 and T2 for each individual country.

A high percentage of crowdworkers indicated that they save or invest the money earned on CrowdFlower, especially in lower income countries. The countries where the highest percentage of workers who chose this response were Venezuela, the Philippines and Indonesia. The USA and Russia had the lowest proportions of workers who reported saving or investing the income from micro tasks.

The USA, Russia and India had the highest proportion of workers who reported spending the money on gifts, while the lowest proportion for this expenditure category was in Venezuela. A moderate percentage of workers stated using the micro-task income for financing their education. This expenditure category was highest in Venezuela, followed by India, Mexico and Indonesia. In most countries, very few workers donate their income from micro tasks to charities, with the exception of India and Indonesia. A significant proportion of workers also stated that they used the money for purposes other than the given categories, especially in the Philippines and in Venezuela.

Figure 20 shows the JS divergences of the answer distributions. As this survey question allowed for multiple answers, we normalized the distributions to sum to one before calculating the JS divergences. We found the largest differences in distribution between Venezuela and the three countries in the high income group as well as Russia. Generally, the countries in the high income group as well as Russia were somewhat similar among each other, and more dissimilar to the countries in the low income group and Venezuela.

Regarding the difference between the time points, Venezuela showed the largest changes. These changes were mostly in the categories basic expenses and education. While in T1 the country with the highest proportion of workers spending the micro task money for basic expenses was the Philippines, in T2 it was Venezuela. The proportion of Venezuelan workers who reported using the micro-task income money use for basic expenses rose from 52.4% in T1 to 62.4% in T2. This is consistent with Venezuelan workers’ increase in relying on CrowdFlower as a primary source of income, as well as their increase in weekly time spent on the platform. The proportion of Venezuelan workers using the money for their education rose from 22% at T1 to 32% at T2.

The second-largest change191919Brazil had a slightly lower JS (0.0025) than Venezuela (0.0028). from T1 to T2 was observed in Brazil. In T2, less Brazilian workers indicated saving or investing their micro-task income, while more Brazilian workers reported spending it on basic expenses and leisure activities. In T2, there was also a lower percentage of workers who reported donating micro-task income to charity than in T1, in all countries of the low income group.

6 Conclusion

The work presented in this paper constitutes the first large scale comparison of crowdworker characteristics at the country level that goes beyond an analysis of the two countries that constitute the majority of workers on MTurk. By shedding light on the country-specific differences of the international crowd workforce, this study complements existing research and contributes to a better understanding of this emerging form of work.

We presented an analysis of the demographic composition of the crowd workforce in ten countries and the centrality of micro-task income in workers’ lives. We based our analysis on two large samples of crowdworkers from ten different countries, collected at two different points in time on the platform CrowdFlower. Our results reveal significant differences in demographic composition, time spent on the platform, reliance on micro-task income as well as use of micro-task income between the different countries. Furthermore, our results show that the characteristics of the workforce in different countries remained, in most cases, largely stable between the two samples collected eight months apart. While there were changes in the answer distributions of certain characteristics in some countries, the average differences between the countries were larger than the average change over time. These results constitute an important step towards a more comprehensive characterization of the international crowd workforce.

Our study has several limitations. While we took great care to account for fluctuations in worker composition (e.g. by the hour of the day or the day of the week) by dividing the starting times of our tasks into different categories, further research on the stability of the different characteristics is needed. Furthermore, due to the nature of micro tasks, our samples are necessarily self-selected. Our samples therefore do not include workers who, for example, exclusively work on repeatable tasks and never accept survey tasks. Lastly, our sample focuses on workers who have sufficient English skills to understand the survey questions. However, this is likely true for the majority of the micro-task workforce on this platform, as workers are expected to understand instructions in English202020The platform’s interface is available exclusively in English. and demand for crowdworkers is driven by Anglophone countries Kuek et al. (2015).

In future work, we plan to analyze the relationship between demographic characteristics and motivational profiles of crowdworkers, using the Multidimensional Crowdworker Motivation Scale Posch et al. (2017). Furthermore, future research will be able to use the data presented in this study in order to compare the demographic composition of the crowd workforce with the composition of the general population, and the general workforce, in different countries. Finally, future research focusing on the examination of factors that cause the differences in crowd workforce composition between countries and over time will further contribute to a better understanding of the phenomenon of crowdwork. This paper is relevant for researchers and practitioners interested in the composition of the international crowd workforce.

References

  • (1)
  • Amazon Mechanical Turk (2016) Amazon Mechanical Turk, . (2016). Amazon Mechanical Turk: Worker Web Site FAQs. http://www.mturk.com/mturk/help?helpPage=worker##how_paid. (2016). Accessed: 2016-12-20.
  • Berg (2016) Berg, J. (2016). Income security in the on-demand economy: findings and policy lessons from a survey of crowdworkers. (2016).
  • Berinsky et al. (2012) Berinsky, A. J, Huber, G. A, and Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com’s Mechanical Turk. Political Analysis 20, 3 (2012), 351–368.
  • Bloomberg News (2016) Bloomberg News, . (2016). Venezuela’s Currency Is Collapsing on the Black Market Again. http://www.bloomberg.com/news/articles/2016-11-01/venezuela-s-currency-is-collapsing-on-the-black-market-again. (2016). Accessed: 2016-12-20.
  • Brabham (2010) Brabham, D. C. (2010). MOVING THE CROWD AT THREADLESS. Information, Communication & Society 13, 8 (2010), 1122–1145. DOI:http://dx.doi.org/10.1080/13691181003624090 
  • Brewer et al. (2016) Brewer, R, Morris, M. R, and Piper, A. M. (2016). Why would anybody do this?: Understanding older adults’ motivations and challenges in crowd work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246–2257.
  • Buhrmester et al. (2011) Buhrmester, M, Kwang, T, and Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6, 1 (2011), 3–5.
  • Destatis (2013) Destatis, . (2013). Einkommens- und Verbrauchsstichprobe: Einnahmen und Ausgaben privater Haushalte. Fachserie 15 des Statistischen Bundesamtes 4 (2013).
  • Difallah et al. (2018) Difallah, D, Filatova, E, and Ipeirotis, P. (2018). Demographics and dynamics of mechanical turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 135–143.
  • European Agency for Health and Safety at Work (2015) European Agency for Health and Safety at Work, . (2015). The future of work: crowdsourcing. https://osha.europa.eu/en/tools-and-publications/publications/future-work-crowdsourcing/view. (2015). Accessed: 2018-10-20.
  • Goodman and Paolacci (2017) Goodman, J. K and Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research 44, 1 (2017), 196–210.
  • Hirth et al. (2011) Hirth, M, Hoßfeld, T, and Tran-Gia, P. (2011). Human cloud as emerging Internet application-anatomy of the microworkers crowdsourcing platform. University of Wurzburg Institute of Computer Science Research Report Series (2011).
  • Huff and Tingley (2015) Huff, C and Tingley, D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics 2, 3 (2015), 2053168015604648.
  • Ipeirotis (2010a) Ipeirotis, P. G. (2010)a. Analyzing the Amazon Mechanical Turk marketplace. ACM Crossroads 17, 2 (2010), 16–21. DOI:http://dx.doi.org/10.1145/1869086.1869094 
  • Ipeirotis (2010b) Ipeirotis, P. G. (2010)b. Demographics of mechanical turk. CeDER Working Papers (2010).
  • Kazai et al. (2012) Kazai, G, Kamps, J, and Milic-Frayling, N. (2012). The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 2583–2586.
  • Kuek et al. (2015) Kuek, S. C, Paradi-Guilford, C, Fayomi, T, Imaizumi, S, Ipeirotis, P, Pina, P, Singh, M, and others, . (2015). The global opportunity in online outsourcing. Technical Report. The World Bank.
  • Lin (1991) Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145–151.
  • Martin et al. (2017) Martin, D, Carpendale, S, Gupta, N, Hoßfeld, T, Naderi, B, Redi, J, Siahaan, E, and Wechsung, I. (2017). Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing. In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Springer, 27–69.
  • Naderi (2018) Naderi, B. (2018). Who are the Crowdworkers? In Motivation of Workers on Microtask Crowdsourcing Platforms. Springer, 17–27.
  • Paolacci and Chandler (2014) Paolacci, G and Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science 23, 3 (2014), 184–188.
  • Paolacci et al. (2010) Paolacci, G, Chandler, J, and Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk. Judgment and Decision making 5, 5 (2010), 411–419.
  • Pavlick et al. (2014) Pavlick, E, Post, M, Irvine, A, Kachaev, D, and Callison-Burch, C. (2014). The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics 2 (2014), 79–92.
  • Peer et al. (2017) Peer, E, Brandimarte, L, Samat, S, and Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70 (2017), 153–163.
  • Posch et al. (2017) Posch, L, Bleier, A, and Strohmaier, M. (2017). Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Motivation Scale. CoRR https://arxiv.org/abs/1702.01661 (2017).
  • Ross et al. (2010) Ross, J, Irani, L, Silberman, M. S, Zaldivar, A, and Tomlinson, B. (2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume, Atlanta, Georgia, USA, April 10-15, 2010, Elizabeth D. Mynatt, Don Schoner, Geraldine Fitzpatrick, Scott E. Hudson, W. Keith Edwards, and Tom Rodden (Eds.). ACM, 2863–2872. DOI:http://dx.doi.org/10.1145/1753846.1753873 
  • Ross et al. (2009) Ross, J, Zaldivar, A, Irani, L, and Tomlinson, B. (2009). Who are the turkers? worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009).
  • Shapiro et al. (2013) Shapiro, D. N, Chandler, J, and Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science 1, 2 (2013), 213–220.
  • United Nations Department Of Economic And Social Affairs - Statistics Division (2000) United Nations Department Of Economic And Social Affairs - Statistics Division, . (2000). Classifications of expenditure according to purpose. https://unstats.un.org/unsd/publication/SeriesM/SeriesM_84E.pdf. (2000). Accessed: 2018-12-02.
  • U.S. Bureau of Labor Statistics (2018) U.S. Bureau of Labor Statistics, . (2018). Consumer Expenditure Surveys - CE Survey Materials. https://www.bls.gov/cex/csxsurveyforms.htm. (2018). Accessed: 2018-12-02.
  • Weinberg et al. (2014) Weinberg, J. D, Freese, J, and McElhattan, D. (2014). Comparing Data Characteristics and Results of an Online Factorial Survey between a Population-based and a Crowdsource-recruited Sample. Sociological Science 1 (2014).