How many labelers do you have? A closer look at gold-standard labels

06/24/2022
by   Chen Cheng, et al.
0

The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of “gold-standard.”. We question the wisdom of this pipeline by developing a (stylized) theoretical model of this process and analyzing its statistical consequences, showing how access to non-aggregated label information can make training well-calibrated models easier or – in some cases – even feasible, whereas it is impossible with only gold-standard labels. The entire story, however, is subtle, and the contrasts between aggregated and fuller label information depend on the particulars of the problem, where estimators that use aggregated information exhibit robust but slower rates of convergence, while estimators that can effectively leverage all labels converge more quickly if they have fidelity to (or can learn) the true labeling process. The theory we develop in the stylized model makes several predictions for real-world datasets, including when non-aggregate labels should improve learning performance, which we test to corroborate the validity of our predictions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/01/2020

Extreme Multi-label Classification from Aggregated Labels

Extreme multi-label classification (XMC) is the problem of finding the r...
research
09/12/2017

Learning with Bounded Instance- and Label-dependent Label Noise

Instance- and label-dependent label noise (ILN) is widely existed in rea...
research
01/13/2013

Crowd Labeling: a survey

Recently, there has been a burst in the number of research projects on h...
research
12/13/2017

Ballpark Crowdsourcing: The Wisdom of Rough Group Comparisons

Crowdsourcing has become a popular method for collecting labeled trainin...
research
04/08/2022

Decomposition-based Generation Process for Instance-Dependent Partial Label Learning

Partial label learning (PLL) is a typical weakly supervised learning pro...
research
06/15/2020

Learnability with Indirect Supervision Signals

Learning from indirect supervision signals is important in real-world AI...
research
11/19/2022

A Light-weight, Effective and Efficient Model for Label Aggregation in Crowdsourcing

Due to the noises in crowdsourced labels, label aggregation (LA) has eme...

Please sign up or login with your details

Forgot password? Click here to reset