DeepAI AI Chat
Log In Sign Up

Stratifying Online Field Experiments Using the Pigeonhole Design

by   Jinglong Zhao, et al.
Boston University

Practitioners and academics have long appreciated the benefits that experimentation brings to firms. For online web-facing firms, however, it still remains challenging in handling heterogeneity when experimental units arrive sequentially in online field experiments. In this paper, we study a novel online experimental design problem, which we refer to as the "Online Stratification Problem." In this problem, experimental units with heterogeneous covariate information arrive sequentially and must be immediately assigned into either the control or the treatment group, with an objective of minimizing the total discrepancy, which is defined as the minimum weight perfect matching between the two groups. To solve this problem, we propose a novel experimental design approach, which we refer to as the "Pigeonhole Design." The pigeonhole design first partitions the covariate space into smaller spaces, which we refer to as pigeonholes, and then, when the experimental units arrive at each pigeonhole, balances the number of control and treatment units for each pigeonhole. We analyze the theoretical performance of the pigeonhole design and show its effectiveness by comparing against two well-known benchmark designs: the match-pair design and the completely randomized design. We conduct extensive simulations to study the numerical performance of the different designs and conclude with practical suggestions.


page 1

page 2

page 3

page 4


Synthetic Controls for Experimental Design

This article studies experimental design in settings where the experimen...

Trustworthy Online Marketplace Experimentation with Budget-split Design

Online experimentation, also known as A/B testing, is the gold standard ...

A Discrepancy-Based Design for A/B Testing Experiments

The aim of this paper is to introduce a new design of experiment method ...

Inference for Two-stage Experiments under Covariate-Adaptive Randomization

This paper studies inference in two-stage randomized experiments with co...

Design and Analysis of Switchback Experiments

In switchback experiments, a firm sequentially exposes an experimental u...

Design Mining Microbial Fuel Cell Cascades

Microbial fuel cells (MFCs) perform wastewater treatment and electricity...

Trimmed Match Design for Randomized Paired Geo Experiments

How to measure the incremental Return On Ad Spend (iROAS) is a fundament...