Statistical Inference for Diagnostic Test Accuracy Studies with Multiple Comparisons

by   Max Westphal, et al.

Diagnostic accuracy studies assess sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test is usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regards to statistical error rates in least-favorable and realistic scenarios. This covers parametric and nonparamtric methods and one Bayesian approach. All methods have been implemented in the new open-source R package DTAmc which allows to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT, Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power.



There are no comments yet.


page 7

page 10


A multiple testing framework for diagnostic accuracy studies with co-primary endpoints

Major advances have been made regarding the utilization of artificial in...

Efficient and powerful equivalency test on combined mean and variance with application to diagnostic device comparison studies

In medical device comparison studies, equivalency test is commonly used ...

How to analyze data in a factorial design? An extensive simulation study

Factorial designs are frequently used in different fields of science, e....

Familywise Error Rate Controlling Procedures for Discrete Data

In applications such as clinical safety analysis, the data of the experi...

Robust Spatial Extent Inference with a Semiparametric Bootstrap Joint Testing Procedure

Spatial extent inference (SEI) is widely used across neuroimaging modali...

Accuracy, Repeatability, and Reproducibility of Firearm Comparisons Part 1: Accuracy

Researchers at the Ames Laboratory-USDOE and the Federal Bureau of Inves...

Bayesian Multivariate Probability of Success Using Historical Data with Strict Control of Family-wise Error Rate

Given the cost and duration of phase III and phase IV clinical trials, t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.