When Crowdsourcing Meets Mobile Sensing: A Social Network Perspective

08/03/2015 ∙ by Pin-Yu Chen, et al. ∙ University of Michigan Amazon National Taiwan University of Science and Technology Georgia Institute of Technology 0

Mobile sensing is an emerging technology that utilizes agent-participatory data for decision making or state estimation, including multimedia applications. This article investigates the structure of mobile sensing schemes and introduces crowdsourcing methods for mobile sensing. Inspired by social network, one can establish trust among participatory agents to leverage the wisdom of crowds for mobile sensing. A prototype of social network inspired mobile multimedia and sensing application is presented for illustrative purpose. Numerical experiments on real-world datasets show improved performance of mobile sensing via crowdsourcing. Challenges for mobile sensing with respect to Internet layers are discussed.



There are no comments yet.


page 1

page 3

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Wireless sensor network (WSN) explores the avenues to collect and use information from the physcial world by deploying low-cost tiny sensor nodes on the ground, in the air, under water, on bodies, in vehicles, and inside buildings. With sensing, processing, and communication capabilities, networked sensor nodes cooperatively collect information on entities of interest and WSNs have emerged as a promising technology with numerous and various applications. As shown in Figure 1, sensor nodes locally collect information and then forward the sensed result over a wireless medium to a remote static sink, where it is fused and analyzed in order to determine the global status of the sensed area. In order to successfully gather sufficient information, a static sink could send a mobile agent to collect data from individual sensor nodes by following a trajectory spanning all the nodes (see Figure 1).

Fig. 1: Evolution from Wireless Sensor Networks to Mobile Sensing.

To accomplish large-scale sensing, WSN evolves not only at the sink side (such as mobile agents), but also at the sensor node side. Mature mobile networks consisted of mobile devices with advanced processing and communication capabilities become a possible sensing infrastructure of WSN. By exploiting the rich set of embedded sensors (such as camera, gyroscope, GPS, accelerator, light sensor and digital compass) on mobile phones as the sensor nodes, a new paradigm of WSN is realized, which is known as mobile sensing [1, 2, 3, 4, 5, 6, 7, 8, 9]. As shown in Figure 1, mobile sensing utilizes crowdsensed information for data analysis and decision making due to penetration of mobile devices as well as human mobility and ubiquity. It relies on the wisdom of crowds [10] to successfully infer the information of interest and accomplish its tasks. The data from the mobile crowds (e.g., users, sensors, robots, e.t.c.) can be either numerical or categorical, depending on applications. Examples of crowdsensed data include numerical environmental measurements such as temperature and air conditions [3, 5], personal activities such as daily life patterns and events [2], interactions among people such as crowd density [7] and common interests [6], categorical recommendations such as ratings for nearby restaurants [1], and user experience/quality feedback of wireless mobile multimedia applications [1].

It is worth noting that many multimedia applications lie within the scope of mobile sensing, since extracting and analyzing the information sensed or generated from the crowds is one of the core goals for many multimedia applications, in order to attract users’ attention. Better prediction of users’ interest leads to longer multimedia stickiness and hence more revenues can be expected. Modern multimedia applications often pull user-centric information from the crowds and offer personalized contents (e.g., next video to watch). Typically, location and social network information are widely used for targeted advertisement and recommendation. Therefore, the major challenge is to efficiently and accurately extract user-centric information from the crowds and identify users of high similarity for improved content delivery.

In recent years, many machine learning tasks and business models have leveraged the wisdom of crowds to acquire crowdsourced data for discriminating unknown objects. The website

Galaxy Zoo

asks visitors to help classify the shapes of galaxies, and the website

Stardust@home asks visitors to help detect interstellar dust particles in astronomical images. Business models such as Amazon Mechanical Turk (MTurk) and CrowdFlower provide crowdsourcing services with low prices. For MTurk, a minimum of 0.01 US dollar111https://requester.mturk.com/pricing is paid to a labeler/worker when he/she makes a click (i.e., generates a label) for an item. Despite the low costs for acquiring crowdsourced data, one of the major challenges of mobile sensing roots in dealing with noisy and potentially erroneous data [11, 4, 8]. These undesired data can originate from environmental/object uncertainties (e.g., channel noise and difficulty of object discrimination) or user intentions (e.g., fraudulent recommendations and irresponsible user clicks). Consequently, identifying trustworthy data and reliable agents becomes an essential task in mobile sensing [4, 5, 8].

Utilizing the concept of trust in social network, this article proposes trust-based data analysis approaches for crowdsourcing in mobile sensing schemes. In addition to scraping trustworthy data from the bulk, these approaches aim to identifying reliable agents for performance enhancement. A weight of trust is built upon the reliability of each agent for mobile sensing via limited number of training queries. For spectrum sensing, these approaches can be implemented by broadcasting reference signals for reception power calibration. For annotation, these approaches can be implemented by uploading some items with known answers. For multimedia, these approaches can be associated with user behaviors based on the provided contents.

This article summarizes mobile sensing network paradigms and introduces several trust-based crowdsourcing methods for mobile sensing. We also illustrate a prototype application of social network inspired mobile multimedia and sensing scheme. Numerical experiments on real-world datasets show that mobile sensing can benefit from crowdsourcing for improved performance. Potential challenges of social network based mobile sensing with respect to mobile multimedia Internet layers are discussed. This article therefore sheds new light on integration of social network and mobile sensing, and applications therein.

Ii Mobile Sensing Paradigms

In mobile sensing, people share and distribute sensed information via physical proximity or social relations over portable sensors. As illustrated in Figure 2, a mobile user plays the roles of both querier and collector, who respectively requests and provides information. A querier can simultaneously be a collector if he/she also participates in mobile sensing. Network structures of mobile sensing can be classified into two categories, namely the direct and indirect paradigms, which are described as follows.

Fig. 2: Network architecture of mobile sensing.

Ii-a Direct mobile sensing paradigm

A direct mobile sensing paradigm involves direct communication between a querier and crowds (i.e., the collectors). Typically, it is achieved by adopting current device-to-device (D2D) communication technologies, such as Wi-Fi Direct, ZigBee, Bluetooth Low Energy (BLE), or near field communication (NFC). In such a case, the store-carry-forward behavior facilitates information delivery in an ad hoc fashion. That is, sensed information could be stored in a sensor node in the absence of immediate connectivity to any other node, and could be relayed to other sensor nodes at encounters. Examples include:

  • Proximity sensing in mobile social networks (MSN): It supports social platforms among physically proximate mobile users. For instance, one can simply scan the environment for discoverable Bluetooth devices to analyze crowd density and crowd flow direction [7]. By exploiting P2P communications, one can further make new social interactions with nearby devices. A popular example is sensing “potential friends with similar interests nearby”. To enjoy such new activities, mobile users have to provide their own interests for profile matching by broadcasting their personal profiles to all nearby users, and then comparing their personal profiles and other users’ profiles for friend matching [6].

  • Cooperative spectrum sensing in cognitive radio networks (CRN): Unlicensed secondary users (SUs) sense the surrounding environment and exploit spectrum holes unoccupied by licensed primary users (PUs) for secondary transmission with minimal interference to PUs. To achieve better spectrum management and to enhance radio resource utilization, a querier could exploit observations on local spectrum vacancy from surrounding SUs (i.e., crowds). The empowerment of cooperative spectrum sensing improves the throughput of wireless communications and reduces potential interference among heterogeneous systems.

Ii-B Indirect mobile sensing paradigm

In this paradigm a querier and crowds are indirectly connected through a communication system in a centralized or a distributed fashion. Typically, access points in WLAN and a base station in cellular network or WiMax are exploited as communication paths in the former case. In the latter case, a querier/collectors could download/upload data from/to nearby relays via localized communication technologies such as wifi-direct, BLE or NFC. Examples include:

  • Environmental measurements: Collectors provide local measurements (e.g., temperatures, air pollution indexes, e.t.c.) to a querier via an existing cellular infrastructure for event detection or state estimation. Consequently, the current environment can be understood and be improved [3, 5]. For example, PEIR project [3] exploits sensors in mobile phones to build a system that tracks the impact of individual actions on carbon emissions.

  • Personal activity sharing: A collector shares his/her daily life patterns, activities (such as sports), health (such as heart rate, blood pressure) with his/her friends using online social networks. For example, by automatically classifying events in people’s lives via sensors on mobile phones, CenceMe [2] enables selective event sharing among friends using Facebook or Twitter.

  • Online recommendation: Crowds (e.g., data collected from proximal users or users of high similarity) provide recommendations to a user-centric query, such as the best seafood restaurant within 2 miles, or the next video to watch for multimedia applications. For example, Micro-Blog [1] encourages user to record multimedia blogs manually or automatically (via sensors). Moreover, the blogs from collectors in the same area are integrated to enrich the contents. Consequently, a querier can browse multimedia blogs at a selected region for relevant information.

  • Annotation: Crowds (e.g., machines, people, e.t.c.) annotate labels, such as scenery labels for a picture or comments and interactions for multimedia contents, for an item inquired by a user. One typical example is the Amazon Mechanical Turk (MTurk) service.

Fig. 3: LifeTie mobile sensing system - physical device and its mobile multimedia interface.

Iii Life Tie: A Prototype Social Network Inspired Mobile Multimedia and Sensing Application

For further illustration, in this section we introduce a social network inspired mobile multimedia and sensing application. In Taiwan, mountainous areas are hikers’ paradise. Hikers are used to tie trail marking ribbons on trees for direction guidance. It is a matter of life and death to clearly know one’s own location, especially at night time. However, trail marking ribbons have several disadvantages such as misinterpretation, lack of detailed information, and environmental pollution caused by overuse. To ensure hikers’ saftey while overcoming the aforementioned drawbacks, we propose a mobile sensing system named “LifeTie”. Unlike traditional WSN deployed in mountains for wildlife tracking and ecological monitoring, LifeTie acts as an annotation platform where users can exchange their sensing results. The integration of tail marking ribbon and NFC technology replaces the traditional marking method with smartphone APP and achieves the purpose of information exchange, rescue, and search.

The main concept of LifeTie is to exploit NFC tags as the enabler for information exchange among people. To achieve that, NFC tags shall be attached around a mountain, thus creating an infrastructure. Hikers can trigger NFC tags nearby (typical in the range of only a few inches) using their NFC enabled mobile phones. Then the NFC enabled mobile phones can read/write data from/to the NFC tags. With embedded memory, LiftTie could handle a vast amount of data, which explores the possibility of multimedia information and advances toward integration of social network and mobile sensing. The prototype of LifeTie is shown in Figure 3. The features and functionalities of LifeTie are summarized as follows.

Iii-a Features

  • Flexible: The shape of LifeTie was inspired by zap-straps. It can be easily tied to tree branches.

  • Recognizable: With the color of fluorescent orange and the addition of reflective stickers, it provides direction guidance even at night time.

  • Reliable: Polypropylene (PP) is used in LifeTie for its bendability and durability. An NFC tag is integrated in the internal parts of a strap to resist extreme weather conditions in mountains.

  • Affordable: The cost of an NFC tag is low.

  • Power-saving: An NFC tag is not powered by electricity. As a result, no replacement for LifeTie is required, and therefore even lower deployment cost can be achieved.

Iii-B Functionalities

  • Navigation and Warning. Via the corresponding APP, hikers can check out LifeTie’s guestbook right after triggering NFC tags. Depending on the current environmental conditions, hikers could leave comments or draw a simple map to make a notification or navigation. Some useful waring icons (such as cliff, snake, wasps, slippery) and guiding icons (such as cave, camp) are provided when drawing the map to enable diverse multimedia contents. The updated surrounding information facilitates the following hikers.

  • Tracking and Rescue. When a hiker is lost in a mountain and finds LifeTie, he/she can check regular comments to see if there is any shelter nearby. Moreover, he/she can leave urgent comments highlighted by red. If he/she can leave such information on several LifeTies, rescuers can easily identify a rough search area according to the positions of deployed LifeTie and timestamps of the comments. As a result, the rescuing operations become more effective and efficient.

Iv Trust-based Crowdsourcing Methods for Mobile Sensing

This section provides an overview of weight (trust) assignment methodologies on agents for crowdsenced data in mobile sensing. The utility of these methods is investigated in Sec. V and the challenges toward practice are addressed in Sec. VI. For crowdsourcing-empowered mobile sensing, a user (or an intermediate system) evaluates a weight of trust for -th agent and fuse information from agents via weighted combination of agents’ observations for the user’s query. As shown in Figure 4, the collected data from the agents can be viewed as a matrix with rows representing agents and columns representing observations associated with queries. The training queries refer to the queries with known answers and they are used for weight evaluation. The number of agents is denoted by .

The final output for mobile sensing is the combination of each agent’s observation multiplied by the associated weight. Here we discuss several crowdsourcing methods involving different weight evaluation approaches. These methods can be classified into two categories, unsupervised and supervised methods, separated by the need of training queries.

Fig. 4: Illustration of crowdsourced data.

Iv-a Unsupervised crowdsourcing methods

  • majority votes: majority votes adopts uniform trust among all agents (i.e., the weight for all ) and selects the observation that most agents agree upon as the final output. This method may lead to poor performance when the majority of agents have incorrect observations or when some observations are maliciously manipulated.

  • probabilistic inference

    : Probabilistic inference method assumes that each observation made by an agent is statistically independent and imposes statistical model to infer the weights from observations. One popular method is the weight evaluation method based on an expectation maximization (EM) algorithm

    [12, 5].

Iv-B Supervised crowdsourcing methods

Supervised crowdsourcing methods aim to find the optimal weight of each agent by solving the optimization problem as


is the vector of weights,

is a regularization function for , and is the regularization parameter for . Here we introduce several supervised crowdsourcing methods.

  • weighted averaging

    weighted averaging is a heuristic weight evaluation approach which assigns weight that is proportional to the accuracy of each agent in the training queries. Let

    be the fraction of correct queries responded by agent . The weight of agent is the normalized accuracy .

  • exponential weighted algorithm: exponential weighted algorithm adopts exponential cost function and zero regularization parameter () and sequentially adjusts the weight of each agent from the training queries. Interested readers can refer to [13] and the references therein for more details.

  • support vector machine

    : support vector machine adopts the Hinge loss function as its cost function and assumes the regularization function

    and positive regularization parameter (i.e.,

    ). Support vector machine aims to find a separating hyperplane that best discriminates the training queries in the data sample space, and the weight of each agent can be determined by the resulting separating hyperplane. Interested readers can refer to

    [13] and the references therein for more details.

  • professional search: Inspired by social networks where problems are often resolved by professionals, professional search aims to assign weights to only a few agents that have outstanding accuracy in the training queries [14]. Professional search adopts the Hinge loss function as its cost function and assumes the regularization function . This regularization function is known as a surrogate function that promotes sparsity in (i.e., most of the weights are zero), and hence the professionals hidden in the crowds can be selected for mobile sensing.

V Numerical Experiments

In this section we use two crowd generated datasets to investigate the performance of the crowdsourcing methods in Sec. IV. For the first dataset, each agent only participates in some fraction of queries and hence it resembles the dynamic participatory nature in mobile sesing. For the second dataset almost every agent responds to each query but none of the agents have correct answers to all queries, which resembles the imperfect sensing capability in mobile sensing. In both scenarios, crowdsourcing methods can improve the query classification performance by identifying trustworthy agents.

For crowdsourcing methods involving a regularization function , we use leave-one-out-cross-validation (LOOCV) approach [13] to determine the optimal regularization parameter , by swiping from to to select the optimal value that leads to minimum training error. One-to-all classification approach is used for multiple (more than two) categorical datasets (e.g., the exam dataset).

V-a Text relevance judgment

The text relevance judgment dataset is provided by the text retrieval conference (TREC) crowdsourcing track in 222https://sites.google.com/site/treccrowd/2011, where agents (participants) are asked to judge the relevance of paragraphs excerpted from a subset of articles with given topics. Each agent then generates an observation, either “relevant” or “irrelevant”, for an article. It is worth mentioning that this dataset is sparse in the sense that in average each agent only read roughly out of articles. For supervised crowdsourcing methods we use roughly ( training queries) of articles to evaluate each agent’s weight. The rest of data samples are used to test the accuracy of crowdsourcing algorithms and the results are summaried in Table I. It is observed that supervised methods can achieve higher accuracy than unsupervised methods via training queries. Also note that support vector machine and professional search outperform other methods since their main objective is to assign more weights on the trustworthy agents/data samples possessing eminent discriminant capability.

Accuracy (%)
TABLE I: The TREC 2011 dataset. Supervised methods attain higher accuracy than unsupervised methods via acquiring a few training queries for weight (trust) assignment.
Accuracy (%)
TABLE II: The science exam dataset. Despite the fact that no students answer all questions correctly, professional search can still achieve more than twice of accuracy than random guess (25% accuracy).

V-B Science exam dataset

The science exam dataset is collected by the authors and it contains questions. Each question has four choices and the correct answer is one of these four choices. There are agents (students) taking the exam and producing observations (their answers). Unlike the TREC dataset, this exam dataset is dense in the sense that almost every student has provided an answer for each question. We use questions as training queries and use the rest to test the accuracy. The results are summarized in Table II. The baseline accuracy (random guess) is . None of the methods can achieve accuracy as high as in the TREC dataset due to the following facts.

  1. The observations of TREC dataset only have two categories, whereas the observations of the exam dataset have four categories, which renders the latter more difficult to be discriminated.

  2. The exam is challenging since majority votes method leads to low accuracy and there is not a student who answers all questions correctly. The best student only has accuracy.

Nonetheless, crowdsourcing methods such as support vector machine and professional search can still attain relatively good accuracy by identifying reliable agents.

Vi Ongoing Challenges: Aspects from Mobile Multimedia Internet Layers

In this section we discuss some ongoing challenges toward integration of social network and mobile sensing, particularly on the aspects of mobile multimedia Internet. Issues corresponding to each layer of mobile multimedia Internet are specified as follows.

Vi-a Application Layer

  • Conflict between privacy and trustworthiness. Although such an integration of social network and mobile sensing is exciting and promising, the collection and sharing of personal information related to human activity introduces the concern of privacy, where the participants are reluctant to reveal any sensitive personal information (such as time, location, pictures, sound, acceleration, and biometric data). As a result, it is crucial to design an approach to collecting sufficient information from participants without violating their privacy [8]. Specifically, authentication shall be supported to identify legal mobile users and adversaries. Moreover, anonymity shall be preserved to hide sensitive information by using technology such as k-anonymous or pseudonyms. These avenues prevent adversary from traversing relationship between users’ contributions and identities.

    In particular, trust-based crowdsourcing methods aim to preserve trustworthiness in harsh environment where malicious participants may deliberately feedback fraudulent data. Obviously, to counteract this effect we need to observe contributions made by each user for a period of time and hence to evaluate his/her trustworthiness. However, it may conflict with the privacy consideration where actual attribute values of a specific user are obscured and the links between multiple contributions from the same user are broken. How to acquire linkability across multiple contributions from the same user while preserving privacy is a challenging issue [5].

  • Data integrity on complicated multimedia content. By manually recording via users or automatically collecting via sensors, a huge amount of information can be retrieved in mobile sensing. The multimedia contents generated from the retrieved materials via application like Micro-Blog [1] contain abundant yet complicated information, which burdens data integrity. Unlike the example raised in Figure 4 where we just need to identify the color of a cloth, multiple parts in one clip or video might lead to distinct conclusion that is highly dependent upon a viewer’s ideology. Extra meta information shall also be included to increase data integrity like user reviews (such as IDMB, Youtube) or user preference (such as Netflix).

  • Incentives of participation. Mobile sensing requires participants to spend their time, attention, and mobile phone’s battery power for contributing data. Obviously, the amount of collected information and the number of participants are proportional to the degree of voluntary. When the amount of collected information is insufficient, the sensing results might not be precise. Thus, mobile sensing needs incentive and mechanism design to encourage people to participate [15].

Vi-B Network Layer

  • Data retrieval in distributed environment. In distributed scenarios (such as proximity sensing, spectrum sensing, or annotation), a querier can only retrieve information from localized collectors, which might lead to biased inference results. To overcome this issue, current researches propose to enable information relay for each individual. Leveraging human mobility and the store-and-forward features, the amount of data collected from crowds grows substantially via information exchange, thereby improving the accuracy of estimation. In addition, in indirect mobile sensing paradigm involving data retrieval and analysis from distributed systems (e.g., data storage servers), distributed computation is known to be one of the big data challenges.

  • The limitations of D2D communications. In the direct sensing paradigm, a querier could communicate with collectors via D2D communications like Wi-Fi Direct, ZigBee, BLE, and NFC. However, current D2D communication technologies typically require manual mutual-authentication when making a connection, which is unfavorable for automatic data collection and device connectivity. Moreover, mobile sensing applications should be capable of integrating the features of different D2D communication technologies (such as transmission ranges, transmission rates, and power consumptions) for ubiquitous multimedia content delivery.

Vi-C Link Layer

  • Unreliable link due to mobility and interference. The mobile nature of users and agents may hinder the performance of mobile sensing due to change in location, environment, and participatory agents. The highly dynamic positions of agents incur ever-changing received interference, thereby affecting the link reliability. To accommodate this effect, a crowdsourcing aided mobile sensing method should possess adaptivity and robustness in such a dynamical situation in order to identity inadequate agent participation and obsolete data collection.

Vi-D Physical Layer

  • Tradeoff between power consumption and sensing accuracy. Apparently, the higher sensing accuracy is attained, the more energy is consumed in mobile devices with limited power due to the increase in data acquisition frequency, which might violate the design rationale of sensing paradigms. Consequently, power consumption fairness and energy-efficient scheduling for participatory devices should be jointly considered and new throughput measures should be studied to balance the tradeoff between power consumption and sensing accuracy.

Vii Conclusion

This article propose to incorporate crowdsourcing methods for mobile sensing and introduces several crowdsourcing methods for evaluating the weight of trust among agents. The direct and indirect mobile sensing network paradigms are discussed. A prototype social network inspired mobile multimedia and sensing application is illustrated toward integration of social network and mobile sensing. Numerical experiments on real-world datasets show that mobile sensing can benefit from crowdsourcing methods for performance improvements. Ongoing challenges of integration of social network and mobile sensing are addressed from the aspects of mobile multimedia Internet layers. This article therefore paves new avenues to various mobile applications and future mobile technology development.


  • [1] E. Gaonkar, J. Li, R. R. Choudhury, L. Cox, and A. Schmidt, “Macro-Blog: Sharing and querying content through mobile phones and social partcipation,” in Proc. ACM MobiSys 2008, June 2008, pp. 174–186.
  • [2] E. Miluzzo, N. D. Lane, K. Fodor, R. Peterson, H. Lu, M. Musolesi, S. B. Eisenman, X. Zheng, and A. T. Campbell, “Sensing meets mobile social networks: the design, implementation and evaluation of the CenceMe application,” in Proc. ACM SenSys 2008, Nov. 2008, pp. 337–350.
  • [3] M. Mun, S. Reddy, K. Shilton, N. Yau, J. Burke, D. Estrin, M. Hansen, E. Howard, R. West, and P. Boda, “PEIR, the personal environmental impact report, as a platform for participatory sensing system research,” in Proc. ACM MobiSys 2009, June 2009, pp. 55–68.
  • [4] R. Ganti, F. Ye, and H. Lei, “Mobile crowdsensing: current state and future challenges,” IEEE Commun. Mag., vol. 49, no. 11, pp. 32–39, Nov. 2011.
  • [5] C. Xiang, P. Yang, C. Tian, Y. Yan, X. Wu, and Y. Liu, “PassFit: Participatory sensing and filtering for identifying truthful urban pollution sources,” IEEE Sensors J., vol. 13, no. 10, pp. 3721–3732, Oct. 2013.
  • [6] R. Zhang, J. Zhang, Y. Zhang, J. Sun, and G. Yan, “Privacy-preserving profile matching for proximity-based mobile social networking,” IEEE J. Sel. Areas Commun., vol. 31, no. 9, pp. 656–668, Sept. 2013.
  • [7] J. Weppner, P. Lukowicz, U. Blanke, and G. Troster, “Participatory Bluetooth scans serving as urban crowd probes,” IEEE Sensors J., vol. 14, no. 12, pp. 4196–4206, Dec. 2014.
  • [8] D. He, S. Chan, and M. Guizani, “User privacy and data trustworthiness in mobile crowd sensing,” IEEE Wireless Commun. Mag., vol. 22, no. 1, pp. 28–34, Feb. 2015.
  • [9] H. Liu, Y. Zhou, and Y. Zhang, “Estimating users’ home and work locations leveraging large-scale crowd-sourced smartphone data,” IEEE Commun. Mag., vol. 53, no. 3, pp. 71–79, Mar. 2015.
  • [10] J. Surowiecki, The Wisdom of Crowds.   Anchor, 2005.
  • [11] R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng, “Cheap and fast-but is it good?: Evaluating non-expert annotations for natural language tasks,” in

    Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)

    , 2008, pp. 254–263.
  • [12] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy, “Learning from crowds,” J. Mach. Learn. Res., vol. 11, pp. 1297–1322, Aug. 2010.
  • [13] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, ser. Springer Series in Statistics, 2001.
  • [14] P.-Y. Chen, C.-W. Lien, F.-J. Chu, P.-S. Ting, and S.-M. Cheng, “Supervised collective classification for crowdsourcing,” arXiv:1507.06682.
  • [15] D. Yang, G. Xue, X. Fang, and J. Tang, “Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing,” in Proc. ACM Mobicom 2012, Aug. 2012, pp. 173–184.