Enhancing statistical inference in psychological research via prospective and retrospective design analysis

09/30/2019
by   Gianmarco Altoè, et al.
0

In the past two decades, psychological science has experienced an unprecedented replicability crisis which uncovered several issues. Among others, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. We build on and further develop an idea proposed by Gelman and Carlin (2014) termed "prospective and retrospective design analysis". Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant), and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). Another important aspect of design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers' awareness during all phases of a research project. We use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups. We examine the case in which the plausible effect size is formalized as a single value, and propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples, we show that even though a design analysis requires big effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/26/2021

A Statistical Inference Framework to Assess Clinical Importance

In clinical research, the effect of a treatment or intervention is widel...
research
02/24/2020

Statistical inference for Axiom A attractors

From the climate system to the effect of the internet on society, chaoti...
research
08/21/2019

Beyond p-values: a phase II dual-criterion design with statistical significance and clinical relevance

Background: Well-designed phase II trials must have acceptable error rat...
research
08/25/2020

Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes

How should scholars evaluate the statistically estimated causal effect o...
research
05/03/2023

Inference at Scale Significance Testing for Large Search and Recommendation Experiments

A number of information retrieval studies have been done to assess which...
research
03/24/2023

Consistent ANOVA-type tests for various effect sizes

Analysis of variance (ANOVA) reveals some disadvantages, such as non-rob...
research
08/17/2020

Presenting the Probabilities of Different Effect Sizes: Towards a Better Understanding and Communication of Statistical Uncertainty

How should social scientists understand and communicate the uncertainty ...

Please sign up or login with your details

Forgot password? Click here to reset