Data Valuation for Medical Imaging Using Shapley Value: Application on A Large-scale Chest X-ray Dataset

by   Siyi Tang, et al.

The reliability of machine learning models can be compromised when trained on low quality data. Many large-scale medical imaging datasets contain low quality labels extracted from sources such as medical reports. Moreover, images within a dataset may have heterogeneous quality due to artifacts and biases arising from equipment or measurement errors. Therefore, algorithms that can automatically identify low quality data are highly desired. In this study, we used data Shapley, a data valuation metric, to quantify the value of training data to the performance of a pneumonia detection algorithm in a large chest X-ray dataset. We characterized the effectiveness of data Shapley in identifying low quality versus valuable data for pneumonia detection. We found that removing training data with high Shapley values decreased the pneumonia detection performance, whereas removing data with low Shapley values improved the model performance. Furthermore, there were more mislabeled examples in low Shapley value data and more true pneumonia cases in high Shapley value data. Our results suggest that low Shapley value indicates mislabeled or poor quality images, whereas high Shapley value indicates data that are valuable for pneumonia detection. Our method can serve as a framework for using data Shapley to denoise large-scale medical imaging datasets.


page 4

page 10

page 28

page 30


Improving Medical Annotation Quality to Decrease Labeling Burden Using Stratified Noisy Cross-Validation

As machine learning has become increasingly applied to medical imaging d...

Deep Feature Learning from a Hospital-Scale Chest X-ray Dataset with Application to TB Detection on a Small-Scale Dataset

The use of ImageNet pre-trained networks is becoming widespread in the m...

Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging

Machine learning models for medical image analysis often suffer from poo...

Explaining Chest X-ray Pathologies in Natural Language

Most deep learning algorithms lack explanations for their predictions, w...

Exploring large scale public medical image datasets

Rationale and Objectives: Medical artificial intelligence systems are de...

Tubular Shape Aware Data Generation for Semantic Segmentation in Medical Imaging

Chest X-ray is one of the most widespread examinations of the human body...

Dataset Distillation for Medical Dataset Sharing

Sharing medical datasets between hospitals is challenging because of the...