Log In Sign Up

Design and Analysis of Switchback Experiments

by   Iavor Bojinov, et al.

In switchback experiments, a firm sequentially exposes an experimental unit to a random treatment, measures its response, and repeats the procedure for several periods to determine which treatment leads to the best outcome. Although practitioners have widely adopted this experimental design technique, the development of its theoretical properties and the derivation of optimal design procedures have been, to the best of our knowledge, elusive. In this paper, we address these limitations by establishing the necessary results to ensure that practitioners can apply this powerful class of experiments with minimal assumptions. Our main result is the derivation of the optimal design of switchback experiments under a range of different assumptions on the order of carryover effect - that is, the length of time a treatment persists in impacting the outcome. We cast the experimental design problem as a minimax discrete robust optimization problem, identify the worst-case adversarial strategy, establish structural results for the optimal design, and finally solve the problem via a continuous relaxation. For the optimal design, we derive two approaches for performing inference after running the experiment. The first provides exact randomization based p-values and the second uses a finite population central limit theorem to conduct conservative hypothesis tests and build confidence intervals. We further provide theoretical results for our inferential procedures when the order of the carryover effect is misspecified. For firms that possess the capability to run multiple switchback experiments, we also provide a data-driven strategy to identify the likely order of carryover effect. To study the empirical properties of our results, we conduct extensive simulations. We conclude the paper by providing some practical suggestions.


page 1

page 2

page 3

page 4


Regression-adjusted average treatment effect estimates in stratified and sequentially randomized experiments

Stratified and sequentially randomized experiments are widely used in fi...

Adaptive A/B Tests and Simultaneous Treatment Parameter Optimization

Constructing asymptotically valid confidence intervals through a valid c...

Design-Based Confidence Sequences for Anytime-valid Causal Inference

Many organizations run thousands of randomized experiments, or A/B tests...

Stratifying Online Field Experiments Using the Pigeonhole Design

Practitioners and academics have long appreciated the benefits that expe...

Optimal Experimental Design for Staggered Rollouts

Experimentation has become an increasingly prevalent tool for guiding po...

Synthetic Principal Component Design: Fast Covariate Balancing with Synthetic Controls

The optimal design of experiments typically involves solving an NP-hard ...