Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data

01/04/2017
by   Jianbo Ye, et al.
0

We proposed a probabilistic approach to joint modeling of participants' reliability and humans' regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously; and regularity measures how often a human will agree with other seriously-entered responses coming from a targeted population. Crowdsourcing-based studies or experiments, which rely on human self-reported affect, pose additional challenges as compared with typical crowdsourcing studies that attempt to acquire concrete non-affective labels of objects. The reliability of participants has been massively pursued for typical non-affective crowdsourcing studies, whereas the regularity of humans in an affective experiment in its own right has not been thoroughly considered. It has been often observed that different individuals exhibit different feelings on the same test question, which does not have a sole correct response in the first place. High reliability of responses from one individual thus cannot conclusively result in high consensus across individuals. Instead, globally testing consensus of a population is of interest to investigators. Built upon the agreement multigraph among tasks and workers, our probabilistic model differentiates subject regularity from population reliability. We demonstrate the method's effectiveness for in-depth robust analysis of large-scale crowdsourced affective data, including emotion and aesthetic assessments collected by presenting visual stimuli to human subjects.

READ FULL TEXT

page 3

page 9

page 14

research
02/25/2023

Mitigating Observation Biases in Crowdsourced Label Aggregation

Crowdsourcing has been widely used to efficiently obtain labeled dataset...
research
09/30/2016

Characterization of experts in crowdsourcing platforms

Crowdsourcing platforms enable to propose simple human intelligence task...
research
02/05/2023

Crowdsourcing Utilizing Subgroup Structure of Latent Factor Modeling

Crowdsourcing has emerged as an alternative solution for collecting larg...
research
06/07/2023

Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3's personality instruments results

To assess the potential applications and limitations of chatbot GPT-3 Da...
research
03/24/2022

k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

Since the inception of crowdsourcing, aggregation has been a common stra...
research
09/10/2019

Investigating Crowdsourcing to Generate Distractors for Multiple-Choice Assessments

We present and analyze results from a pilot study that explores how crow...

Please sign up or login with your details

Forgot password? Click here to reset