Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes
share
How should scholars evaluate the statistically estimated causal effect of a policy intervention? I point out three limitations in the conventional practice. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks variation in plausible effect sizes. Third, the criterion of substantive significance is rarely explained or justified. To address these issues, I propose an original Bayesian decision-theoretic model for binary outcomes. I incorporate the posterior distribution of a causal effect reducing the likelihood of an undesirable event, into a loss function over the cost of a policy to realize the effect and the cost of the event. The model can use an effect size of interest other than the standard point estimate, and the probability of this effect as a continuous measure of uncertainty. It then presents approximately up to what ratio between the two costs an expected loss remains smaller if the policy is implemented than if not. I exemplify my model through three applications and provide an R package for easy implementation.
READ FULL TEXT