Why "Redefining Statistical Significance" Will Not Improve Reproducibility and Could Make the Replication Crisis Worse

11/21/2017
by   Harry Crane, et al.
0

A recent proposal to "redefine statistical significance" (Benjamin, et al. Nature Human Behaviour, 2017) claims that false positive rates "would immediately improve" by factors greater than two and replication rates would double simply by changing the conventional cutoff for 'statistical significance' from P<0.05 to P<0.005. I analyze the veracity of these claims, focusing especially on how Benjamin, et al neglect the effects of P-hacking in assessing the impact of their proposal. My analysis shows that once P-hacking is accounted for the perceived benefits of the lower threshold all but disappear, prompting two main conclusions: (i) The claimed improvements to false positive rate and replication rate in Benjamin, et al (2017) are exaggerated and misleading. (ii) There are plausible scenarios under which the lower cutoff will make the replication crisis worse.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/19/2021

A Proposed Hybrid Effect Size Plus p-Value Criterion: A Replication of Goodman et al. (2019)

In a recent simulation study, Goodman et al. (2019) compare several meth...
research
06/01/2023

TriSig: Assessing the statistical significance of triclusters

Tensor data analysis allows researchers to uncover novel patterns and re...
research
03/20/2019

Statistical Methods for Replicability Assessment

Large-scale replication studies like the Reproducibility Project: Psycho...
research
05/10/2020

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

The last decade saw the emergence of systematic large-scale replication ...
research
05/19/2020

Identifying Statistical Bias in Dataset Replication

Dataset replication is a useful tool for assessing whether improvements ...
research
05/25/2021

Reproducibility Companion Paper: Knowledge Enhanced Neural Fashion Trend Forecasting

This companion paper supports the replication of the fashion trend forec...

Please sign up or login with your details

Forgot password? Click here to reset