DeepAI AI Chat
Log In Sign Up

Needle in a Haystack: An Analysis of Finding Qualified Workers on MTurk for Summarization

by   Lining Zhang, et al.

The acquisition of high-quality human annotations through crowdsourcing platforms like Amazon Mechanical Turk (MTurk) is more challenging than expected. The annotation quality might be affected by various aspects like annotation instructions, Human Intelligence Task (HIT) design, and wages paid to annotators, etc. To avoid potentially low-quality annotations which could mislead the evaluation of automatic summarization system outputs, we investigate the recruitment of high-quality MTurk workers via a three-step qualification pipeline. We show that we can successfully filter out bad workers before they carry out the evaluations and obtain high-quality annotations while optimizing the use of resources. This paper can serve as basis for the recruitment of qualified annotators in other challenging annotation tasks.


page 6

page 11


Millionaire: A Hint-guided Approach for Crowdsourcing

Modern machine learning is migrating to the era of complex models, which...

Re-Examining Human Annotations for Interpretable NLP

Explanation methods in Interpretable NLP often explain the model's decis...

Role of Intrinsic Motivation in User Interface Design to Enhance Worker Performance in Amazon MTurk

Biologists and scientists have been tackling the problem of marine life ...

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

Qualification tests in crowdsourcing are often used to pre-filter worker...

A Survey of NLP-Related Crowdsourcing HITs: what works and what does not

Crowdsourcing requesters on Amazon Mechanical Turk (AMT) have raised que...

In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers

We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsour...

DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit

Dialog system developers need high-quality data to train, fine-tune and ...