An Approach for Noisy, Crowdsourced Datasets Utilizing Ensemble Modeling, 'Human Softmax' Distributions, and Entropic Measures of Uncertainty

10/28/2022
by   Graham West, et al.
0

Noisy, crowdsourced image datasets prove challenging, even for the best neural networks. Two issues which complicate classification on such datasets are class imbalance and ground-truth uncertainty in labeling. The AL-ALL and AL-PUB datasets-consisting of tightly cropped, individual characters from images of ancient Greek papyri are strongly affected by both issues. The application of ensemble modeling to such a dataset can help identify images where the ground-truth is questionable and quantify the trustworthiness of those samples. We apply stacked generalization consisting of nearly identical ResNets: one utilizing cross-entropy (CXE) and the other Kullback-Liebler Divergence (KLD). The CXE network uses standard labeling drawn from the crowdsourced consensus. In contrast, the KLD network uses probabilistic labeling for each image derived from the distribution of crowdsourced annotations. We refer to this labeling as the Human Softmax (HSM) distribution. For our ensemble model, we apply a k-nearest neighbors model to the outputs of the CXE and KLD networks. Individually, the ResNet models have approximately 93 perform an analysis of the Shannon entropy of the various models' output distributions to measure classification uncertainty.

READ FULL TEXT

page 2

page 3

page 5

research
03/13/2023

Collision Cross-entropy and EM Algorithm for Self-labeled Classification

We propose "collision cross-entropy" as a robust alternative to the Shan...
research
07/05/2023

Evaluating AI systems under uncertain ground truth: a case study in dermatology

For safety, AI systems in health undergo thorough evaluations before dep...
research
06/26/2021

Midpoint Regularization: from High Uncertainty Training to Conservative Classification

Label Smoothing (LS) improves model generalization through penalizing mo...
research
09/04/2023

Uncertainty in AI: Evaluating Deep Neural Networks on Out-of-Distribution Images

As AI models are increasingly deployed in critical applications, ensurin...
research
03/01/2020

GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling

In the big data era, data labeling can be obtained through crowdsourcing...
research
06/02/2021

Survey Equivalence: A Procedure for Measuring Classifier Accuracy Against Human Labels

In many classification tasks, the ground truth is either noisy or subjec...
research
01/30/2022

OpTopNET: A Learning Optimal Topology Synthesizer for Ad-hoc Robot Networks

In this paper, we synthesize a machine-learning stacked ensemble model a...

Please sign up or login with your details

Forgot password? Click here to reset