DeepAI AI Chat
Log In Sign Up

Design-Based Confidence Sequences for Anytime-valid Causal Inference

by   Dae Woong Ham, et al.

Many organizations run thousands of randomized experiments, or A/B tests, to statistically quantify and detect the impact of product changes. Analysts take these results to augment decision-making around deployment and investment opportunities, making the time it takes to detect an effect a key priority. Often, these experiments are conducted on customers arriving sequentially; however, the analysis is only performed at the end of the study. This is undesirable because strong effects can be detected before the end of the study, which is especially relevant for risk mitigation when the treatment effect is negative. Alternatively, analysts could perform hypotheses tests more frequently and stop the experiment when the estimated causal effect is statistically significant; this practice is often called "peeking." Unfortunately, peeking invalidates the statistical guarantees and quickly leads to a substantial uncontrolled type-1 error. Our paper provides valid confidence sequences from the design-based perspective, where we condition on the full set of potential outcomes and perform inference on the obtained sample. Our design-based confidence sequence accommodates a wide variety of sequential experiments in an assumption-light manner. In particular, we build confidence sequences for 1) the average treatment effect for different individuals arriving sequentially, 2) the reward mean difference in multi-arm bandit settings with adaptive treatment assignments, 3) the contemporaneous treatment effect for single time series experiment with potential carryover effects in the potential outcome, and 4) the average contemporaneous treatment effect in panel experiments. We further provide a variance reduction technique that incorporates modeling assumptions and covariates to reduce the confidence sequence width proportional to how well the analyst can predict the next outcome.


Design-Based Inference for Multi-arm Bandits

Multi-arm bandits are gaining popularity as they enable real-world seque...

Privacy Aware Experiments without Cookies

Consider two brands that want to jointly test alternate web experiences ...

From one environment to many: The problem of replicability of statistical inferences

The environment in which an experiment is conducted is unique to each ex...

Design and Analysis of Switchback Experiments

In switchback experiments, a firm sequentially exposes an experimental u...

Robust Estimation of Data-Dependent Causal Effects based on Observing a Single Time-Series

Consider the case that one observes a single time-series, where at each ...

A Method for Measuring Network Effects of One-to-One Communication Features in Online A/B Tests

A/B testing is an important decision making tool in product development ...

Optimising Individual-Treatment-Effect Using Bandits

Applying causal inference models in areas such as economics, healthcare ...