Modeling Multiple Annotator Expertise in the Semi-Supervised Learning Scenario

by   Yan Yan, et al.

Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration,multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumstances ground-truth may not exist). Semi-supervised learning approaches have shown that utilizing the unlabeled data is often beneficial in these cases. This paper presents a probabilistic semi-supervised model and algorithm that allows for learning from both unlabeled and labeled data in the presence of multiple annotators. We assume that it is known what annotator labeled which data points. The proposed approach produces annotator models that allow us to provide (1) estimates of the true label and (2) annotator variable expertise for both labeled and unlabeled data. We provide numerical comparisons under various scenarios and with respect to standard semi-supervised learning. Experiments showed that the presented approach provides clear advantages over multi-annotator methods that do not use the unlabeled data and over methods that do not use multi-labeler information.



There are no comments yet.



Semi-Supervised learning with Density-Ratio Estimation

In this paper, we study statistical properties of semi-supervised learni...

Data-Centric Semi-Supervised Learning

We study unsupervised data selection for semi-supervised learning (SSL),...

Renal Cell Carcinoma Detection and Subtyping with Minimal Point-Based Annotation in Whole-Slide Images

Obtaining a large amount of labeled data in medical imaging is laborious...

Semi-supervised Learning for Marked Temporal Point Processes

Temporal Point Processes (TPPs) are often used to represent the sequence...

Semi-Supervised Learning for Detecting Human Trafficking

Human trafficking is one of the most atrocious crimes and among the chal...

Lidar-Camera Co-Training for Semi-Supervised Road Detection

Recent advances in the field of machine learning and computer vision hav...

Semi-supervised Ranking Pursuit

We propose a novel sparse preference learning/ranking algorithm. Our alg...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.