Sensitivity Analysis in the Generalization of Experimental Results

02/07/2022
by   Melody Huang, et al.
0

Randomized controlled trials (RCT's) allow researchers to estimate causal effects in an experimental sample with minimal identifying assumptions. However, to generalize or transport a causal effect from an RCT to a target population, researchers must adjust for a set of treatment effect moderators. In practice, it is impossible to know whether the set of moderators has been properly accounted for. In the following paper, I propose a three parameter sensitivity analysis for generalizing or transporting experimental results using weighted estimators, with several advantages over existing methods. First, the framework does not require assumptions on the underlying data generating process for either the experimental sample selection mechanism or treatment effect heterogeneity. Second, I show that the sensitivity parameters are guaranteed to be bounded and propose several tools researchers can use to perform sensitivity analysis: (1) graphical and numerical summaries for researchers to assess how robust a point estimate is to killer confounders; (2) an extreme scenario analysis; and (3) a formal benchmarking approach for researchers to estimate potential sensitivity parameter values using existing data. Finally, I demonstrate that the proposed framework can be easily extended to the class of doubly robust, augmented weighted estimators. The sensitivity analysis framework is applied to a set of Jobs Training Program experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/05/2019

Covariate Selection for Generalizing Experimental Results

Scientists are interested in generalizing causal effects estimated in an...
research
01/12/2023

A Framework for Generalization and Transportation of Causal Estimates Under Covariate Shift

Randomized experiments are an excellent tool for estimating internally v...
research
06/14/2022

Sensitivity Analysis for Survey Weights

Survey weighting allows researchers to account for bias in survey sample...
research
12/30/2022

Sensitivity Analysis with the R^2-calculus

Causal inference necessarily relies upon untestable assumptions; hence, ...
research
04/20/2023

Identification and multiply robust estimation in causal mediation analysis with treatment noncompliance

In experimental and observational studies, there is often interest in un...
research
09/14/2022

Drawing Causal Inferences About Performance Effects in NLP

This article emphasizes that NLP as a science seeks to make inferences a...

Please sign up or login with your details

Forgot password? Click here to reset