DeepAI AI Chat
Log In Sign Up

Nobody of the Crowd: An Empirical Evaluation on Worker Clustering in Topcoder

by   Razieh Saremi, et al.
Concordia University
Stevens Institute of Technology

Context: Software crowdsourcing platforms typically employ extrinsic rewards such as rating or ranking systems to motivate workers. Such rating systems are noisy and only provide limited knowledge about worker's preferences and performance. Goal: The objective of this study is to empirically investigate patterns and effects of worker behavior in software crowdsourcing platforms in order to improve the success and efficiency of software crowdsourcing. Method: First, we create the bipartite network of active workers based on common registration for tasks. Then, we use the Clauset-Newman-Moore graph clustering algorithm to identify developer clusters in the network. Finally, we conduct an empirical evaluation to measure and analyze workers' behavior per identified cluster in the platform by workers' ranking. More specifically, workers' behavior is analyzed based on worker reliability, worker trustworthiness, and worker success as measures for workers' performance, worker efficiency, and worker elasticity to represent workers' preferences, and worker contest, worker confidence, and worker deceitfulness to understand workers' strategies. The empirical study is conducted on more than one year's real-world data from topcoder, one of the leading software crowdsourcing platforms. Results: We identify four clusters of active workers: mixed-ranked, high-ranked, mid-ranked, and low-ranked. Based on statistical analysis, this study can only support that the low ranked group associates with the highest reliable workers with an average reliability of 25 most trustworthy workers with average trustworthiness of 16 These findings are helpful for task requesters to understand preferences and relations among unknown resources in the platform and plan for task success in a more effective and efficient manner in a software crowdsourcing platform.


page 1

page 2

page 3

page 4


Study on Patterns and Effect of Task Diversity in Software Crowdsourcing

Context: The success of software crowdsourcing depends on steady tasks s...

Creativity on Paid Crowdsourcing Platforms

General-purpose crowdsourcing platforms are increasingly being harnessed...

On the Design of Strategic Task Recommendations for Sustainable Crowdsourcing-Based Content Moderation

Crowdsourcing-based content moderation is a platform that hosts content ...

Low Complexity Recruitment for Collaborative Mobile Crowdsourcing Using Graph Neural Networks

Collaborative Mobile crowdsourcing (CMCS) allows entities, e.g., local a...

CrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designs

We present CrowdHub, a tool for running systematic evaluations of task d...

Who wants to cooperate-and why? Attitude and perception of crowd workers in online labor markets

Existing literature and studies predominantly focus on how crowdsource w...

Universal Clustering via Crowdsourcing

Consider unsupervised clustering of objects drawn from a discrete set, t...