DeepAI AI Chat
Log In Sign Up

Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to the Grading of Astrocytoma Tissues

by   Claudia Beleites, et al.

We use partial class memberships in soft classification to model uncertain labelling and mixtures of classes. Partial class memberships are not restricted to predictions, but may also occur in reference labels (ground truth, gold standard diagnosis) for training and validation data. Classifier performance is usually expressed as fractions of the confusion matrix, such as sensitivity, specificity, negative and positive predictive values. We extend this concept to soft classification and discuss the bias and variance properties of the extended performance measures. Ambiguity in reference labels translates to differences between best-case, expected and worst-case performance. We show a second set of measures comparing expected and ideal performance which is closely related to regression performance, namely the root mean squared error RMSE and the mean absolute error MAE. All calculations apply to classical crisp classification as well as to soft classification (partial class memberships and/or one-class classifiers). The proposed performance measures allow to test classifiers with actual borderline cases. In addition, hardening of e.g. posterior probabilities into class labels is not necessary, avoiding the corresponding information loss and increase in variance. We implement the proposed performance measures in the R package "softclassval", which is available from CRAN and at Our reasoning as well as the importance of partial memberships for chemometric classification is illustrated by a real-word application: astrocytoma brain tumor tissue grading (80 patients, 37000 spectra) for finding surgical excision borders. As borderline cases are the actual target of the analytical technique, samples which are diagnosed to be borderline cases must be included in the validation.


page 9

page 15


Harmless label noise and informative soft-labels in supervised classification

Manual labelling of training examples is common practice in supervised l...

Performance measures for classification systems with rejection

Classifiers with rejection are essential in real-world applications wher...

Sensitivity analysis for random measurement error using regression calibration and simulation-extrapolation

Sensitivity analysis for measurement error can be applied in the absence...

DeepDoseNet: A Deep Learning model for 3D Dose Prediction in Radiation Therapy

The DeepDoseNet 3D dose prediction model based on ResNet and Dilated Den...

SoftSeg: Advantages of soft versus binary training for image segmentation

Most image segmentation algorithms are trained on binary masks formulate...

Classification without labels: Learning from mixed samples in high energy physics

Modern machine learning techniques can be used to construct powerful mod...

A Novel ECOC Algorithm with Centroid Distance Based Soft Coding Scheme

In ECOC framework, the ternary coding strategy is widely deployed in cod...