From one environment to many: The problem of replicability of statistical inferences

04/22/2019
by   James J. Higgins, et al.
0

The environment in which an experiment is conducted is unique to each experiment. While the statistical inferences that are drawn from the analysis of experimental data apply only to the environment in which the experiment is conducted, it is almost always the intent of the researcher to apply the results more broadly. The questions then become, will statistically significant results obtained in one environment carry over to others, and if so, how much of a change in environment can be tolerated before inferences are no longer valid? We answer these questions quantitatively by proposing three measures of replicability for statistical inferences: the probability of replicability, the adjusted p-value, and the adjusted confidence interval. Through these measures, we are able to show that the primary factors for replicability of an experiment are the treatment effect size and the standard deviation of the environment by treatment interaction, both of which may be expressed as ratios relative to the standard deviation of experimental error. We show that larger effect sizes and smaller environmental variability allow for experimental results that are more likely to be replicated. Moreover, if environmental effects are not accounted for, replicating a finding may be equivalent to flipping a fair coin, regardless of how well the initial experiment was conducted.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/16/2022

Design-Based Confidence Sequences for Anytime-valid Causal Inference

Many organizations run thousands of randomized experiments, or A/B tests...
research
04/24/2020

The Generalized Oaxaca-Blinder Estimator

After performing a randomized experiment, researchers often use ordinary...
research
02/23/2022

Learning about treatment effects in a new target population under transportability assumptions for relative effect measures

Epidemiologists and applied statisticians often believe that relative ef...
research
01/18/2022

Who Increases Emergency Department Use? New Insights from the Oregon Health Insurance Experiment

We provide new insights into the finding that Medicaid increased emergen...
research
06/02/2023

Priming bias versus post-treatment bias in experimental designs

Conditioning on variables affected by treatment can induce post-treatmen...
research
02/06/2018

How to Make Causal Inferences Using Texts

New text as data techniques offer a great promise: the ability to induct...
research
11/15/2021

Revisiting C.S.Peirce's Experiment: 150 Years Later

An iconoclastic philosopher and polymath, Charles Sanders Peirce (1837-1...

Please sign up or login with your details

Forgot password? Click here to reset