Connecting Simple and Precise P-values to Complex and Ambiguous Realities

04/03/2023
by   Sander Greenland, et al.
0

Mathematics is a limited component of solutions to real-world problems, as it expresses only what is expected to be true if all our assumptions are correct, including implicit assumptions that are omnipresent and often incorrect. Statistical methods are rife with implicit assumptions whose violation can be life-threatening when results from them are used to set policy. Among them are that there is human equipoise or unbiasedness in data generation, management, analysis, and reporting. These assumptions correspond to levels of cooperation, competence, neutrality, and integrity that are absent more often than we would like to believe. Given this harsh reality, we should ask what meaning, if any, we can assign to the P-values, 'statistical significance' declarations, 'confidence' intervals, and posterior probabilities that are used to decide what and how to present (or spin) discussions of analyzed data. By themselves, P-values and CI do not test any hypothesis, nor do they measure the significance of results or the confidence we should have in them. The sense otherwise is an ongoing cultural error perpetuated by large segments of the statistical and research community via misleading terminology. So-called 'inferential' statistics can only become contextually interpretable when derived explicitly from causal stories about the real data generator (such as randomization), and can only become reliable when those stories are based on valid and public documentation of the physical mechanisms that generated the data. Absent these assurances, traditional interpretations of statistical results become pernicious fictions that need to be replaced by far more circumspect descriptions of data and model relations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/18/2019

Semantic and Cognitive Tools to Aid Statistical Inference: Replace Confidence and Significance by Compatibility and Surprise

Researchers often misinterpret and misrepresent statistical outputs. Thi...
research
09/18/2019

To Aid Statistical Inference, Emphasize Unconditional Descriptions of Statistics

We have elsewhere reviewed proposals to reform terminology and improve i...
research
06/05/2022

Inference for Interpretable Machine Learning: Fast, Model-Agnostic Confidence Intervals for Feature Importance

In order to trust machine learning for high-stakes problems, we need mod...
research
09/08/2023

Confidence in Causal Inference under Structure Uncertainty in Linear Causal Models with Equal Variances

Inferring the effect of interventions within complex systems is a fundam...
research
02/17/2020

Are You Sure You're Sure? – Effects of Visual Representation on the Cliff Effect in Statistical Inference

Common reporting styles of statistical results, such as confidence inter...
research
04/28/2020

Showing Your Work Doesn't Always Work

In natural language processing, a recently popular line of work explores...
research
07/11/2020

Frequentism-as-model

Most statisticians are aware that probability models interpreted in a fr...

Please sign up or login with your details

Forgot password? Click here to reset