DeepAI AI Chat
Log In Sign Up

AgreementLearning: An End-to-End Framework for Learning with Multiple Annotators without Groundtruth

by   Chongyang Wang, et al.
University of Cambridge

The annotation of domain experts is important for some medical applications where the objective groundtruth is ambiguous to define, e.g., the rehabilitation for some chronic diseases, and the prescreening of some musculoskeletal abnormalities without further medical examinations. However, improper uses of the annotations may hinder developing reliable models. On one hand, forcing the use of a single groundtruth generated from multiple annotations is less informative for the modeling. On the other hand, feeding the model with all the annotations without proper regularization is noisy given existing disagreements. For such issues, we propose a novel agreement learning framework to tackle the challenge of learning from multiple annotators without objective groundtruth. The framework has two streams, with one stream fitting with the multiple annotators and the other stream learning agreement information between the annotators. In particular, the agreement learning stream produces regularization information to the classifier stream, tuning its decision to be better in line with the agreement between the annotators. The proposed method can be easily plugged to existing backbones developed with majority-voted groundtruth or multiple annotations. Thereon, experiments on two medical datasets demonstrate improved agreement levels with annotators.


page 1

page 2

page 3

page 4


A Study on Agreement in PICO Span Annotations

In evidence-based medicine, relevance of medical literature is determine...

Sparse Probability of Agreement

Measuring inter-annotator agreement is important for annotation tasks, b...

Annotator Consensus Prediction for Medical Image Segmentation with Diffusion Models

A major challenge in the segmentation of medical images is the large int...

MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions

Backchannels, i.e. short interjections of the listener, serve important ...

Evaluation of Medical Image Segmentation Models for Uncertain, Small or Empty Reference Annotations

Performance metrics for medical image segmentation models are used to me...

Learning Agreement from Multi-source Annotations for Medical Image Segmentation

In medical image analysis, it is typical to merge multiple independent a...

Agreement Implies Accuracy for Substitutable Signals

Inspired by Aumann's agreement theorem, Scott Aaronson studied the amoun...