Pitfalls and Potentials in Simulation Studies

03/24/2022
by   Samuel Pawel, et al.
0

Comparative simulation studies are workhorse tools for benchmarking statistical methods, but if not performed transparently they may lead to overoptimistic or misleading conclusions. The current publication requirements adopted by statistics journals do not prevent questionable research practices such as selective reporting. The past years have witnessed numerous suggestions and initiatives to improve on these issues but little progress can be seen to date. In this paper we discuss common questionable research practices which undermine the validity of findings from comparative simulation studies. To illustrate our point, we invent a novel prediction method with no expected performance gain and benchmark it in a pre-registered comparative simulation study. We show how easy it is to make the method appear superior over well-established competitor methods if no protocol is in place and various questionable research practices are employed. Finally, we provide researchers, reviewers, and other academic stakeholders with concrete suggestions for improving the methodological quality of comparative simulation studies, most importantly the need for pre-registered simulation protocols.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/08/2017

Using simulation studies to evaluate statistical methods

Simulation studies are computer experiments which involve creating data ...
research
08/02/2022

On the role of benchmarking data sets and simulations in method comparison studies

Method comparisons are essential to provide recommendations and guidance...
research
11/26/2021

Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments

Statisticians recommend the Design and Analysis of Experiments (DAE) for...
research
06/04/2021

Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results

In recent years, the need for neutral benchmark studies that focus on th...
research
10/08/2021

A Study Protocol for an Instrumental Variables Analysis of the Comparative Effectiveness of two Prostate Cancer Drugs

This paper presents a protocol, or design, for the analysis of a compara...
research
10/15/2020

An Artifact-based Workflow for Finite-Element Simulation Studies

Workflow support typically focuses on single simulation experiments. Thi...
research
01/01/2022

Challenges of sampling and how phylogenetic comparative methods help: With a case study of the Pama-Nyungan laminal contrast

Phylogenetic comparative methods are new in our field and are shrouded, ...

Please sign up or login with your details

Forgot password? Click here to reset