Evaluating Complex Task through Crowdsourcing: Multiple Views Approach

03/30/2017
by   Lingyu Lyu, et al.
0

With the popularity of massive open online courses, grading through crowdsourcing has become a prevalent approach towards large scale classes. However, for getting grades for complex tasks, which require specific skills and efforts for grading, crowdsourcing encounters a restriction of insufficient knowledge of the workers from the crowd. Due to knowledge limitation of the crowd graders, grading based on partial perspectives becomes a big challenge for evaluating complex tasks through crowdsourcing. Especially for those tasks which not only need specific knowledge for grading, but also should be graded as a whole instead of being decomposed into smaller and simpler subtasks. We propose a framework for grading complex tasks via multiple views, which are different grading perspectives defined by experts for the task, to provide uniformity. Aggregation algorithm based on graders variances are used to combine the grades for each view. We also detect bias patterns of the graders, and debias them regarding each view of the task. Bias pattern determines how the behavior is biased among graders, which is detected by a statistical technique. The proposed approach is analyzed on a synthetic data set. We show that our model gives more accurate results compared to the grading approaches without different views and debiasing algorithm.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/07/2021

Open-Set Crowdsourcing using Multiple-Source Transfer Learning

We raise and define a new crowdsourcing scenario, open set crowdsourcing...
research
02/08/2023

Multiview Representation Learning from Crowdsourced Triplet Comparisons

Crowdsourcing has been used to collect data at scale in numerous fields....
research
07/19/2023

LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs

LLMs have shown promise in replicating human-like behavior in crowdsourc...
research
03/07/2019

Accurate inference of crowdsourcing properties when using efficient allocation strategies

Allocation strategies improve the efficiency of crowdsourcing by decreas...
research
11/05/2020

On the impact of predicate complexity in crowdsourced classification tasks

This paper explores and offers guidance on a specific and relevant probl...
research
08/02/2023

Evaluate and Guard the Wisdom of Crowds: Zero Knowledge Proofs for Crowdsourcing Truth Inference

Due to the risks of correctness and security in outsourced cloud computi...
research
05/16/2017

Subjective Knowledge Acquisition and Enrichment Powered By Crowdsourcing

Knowledge bases (KBs) have attracted increasing attention due to its gre...

Please sign up or login with your details

Forgot password? Click here to reset