Toward Effective Automated Content Analysis via Crowdsourcing

01/12/2021
by   Jiele Wu, et al.
0

Many computer scientists use the aggregated answers of online workers to represent ground truth. Prior work has shown that aggregation methods such as majority voting are effective for measuring relatively objective features. For subjective features such as semantic connotation, online workers, known for optimizing their hourly earnings, tend to deteriorate in the quality of their responses as they work longer. In this paper, we aim to address this issue by proposing a quality-aware semantic data annotation system. We observe that with timely feedback on workers' performance quantified by quality scores, better informed online workers can maintain the quality of their labeling throughout an extended period of time. We validate the effectiveness of the proposed annotation system through i) evaluating performance based on an expert-labeled dataset, and ii) demonstrating machine learning tasks that can lead to consistent learning behavior with 70 with our system, researchers can collect high-quality answers of subjective semantic features at a large scale.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/24/2019

Attention-Aware Answers of the Crowd

Crowdsourcing is a relatively economic and efficient solution to collect...
research
04/26/2022

Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?

Crowdsourcing is an online outsourcing mode which can solve the current ...
research
02/12/2018

Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing

The questions in a crowdsourcing task typically exhibit varying degrees ...
research
05/02/2019

Truth Discovery via Proxy Voting

Truth discovery is a general name for a broad range of statistical metho...
research
02/08/2023

AVeCQ: Anonymous Verifiable Crowdsourcing with Worker Qualities

In crowdsourcing systems, requesters publish tasks, and interested worke...
research
03/07/2022

Towards Automated Real-time Evaluation in Text-based Counseling

Automated real-time evaluation of counselor-client interaction is import...
research
05/17/2020

DEXA: Supporting Non-Expert Annotators with Dynamic Examples from Experts

The success of crowdsourcing based annotation of text corpora depends on...

Please sign up or login with your details

Forgot password? Click here to reset