Log In Sign Up

Learning from Crowds by Modeling Common Confusions

by   Zhendong Chu, et al.

Crowdsourcing provides a practical way to obtain large amounts of labeled data at a low cost. However, the annotation quality of annotators varies considerably, which imposes new challenges in learning a high-quality model from the crowdsourced annotations. In this work, we provide a new perspective to decompose annotation noise into common noise and individual noise and differentiate the source of confusion based on instance difficulty and annotator expertise on a per-instance-annotator basis. We realize this new crowdsourcing model by an end-to-end learning solution with two types of noise adaptation layers: one is shared across annotators to capture their commonly shared confusions, and the other one is pertaining to each annotator to realize individual confusion. To recognize the source of noise in each annotation, we use an auxiliary network to choose the two noise adaptation layers with respect to both instances and annotators. Extensive experiments on both synthesized and real-world benchmarks demonstrate the effectiveness of our proposed common noise adaptation solution.


page 1

page 2

page 3

page 4


Improve Learning from Crowds via Generative Augmentation

Crowdsourcing provides an efficient label collection schema for supervis...

Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction

Modern NLP systems require high-quality annotated data. In specialized d...

Crowdsourcing in Computer Vision

Computer vision systems require large amounts of manually annotated data...

Confidence Scores Make Instance-dependent Label-noise Learning Possible

Learning with noisy labels has drawn a lot of attention. In this area, m...

Multi-Label Annotation Aggregation in Crowdsourcing

As a means of human-based computation, crowdsourcing has been widely use...

Overview of Annotation Creation: Processes & Tools

Creating linguistic annotations requires more than just a reliable annot...