A Discrepancy-Based Design for A/B Testing Experiments

01/25/2019
by   Yiou Li, et al.
0

The aim of this paper is to introduce a new design of experiment method for A/B tests in order to balance the covariate information in all treatment groups. A/B tests (or "A/B/n tests") refer to the experiments and the corresponding inference on the treatment effect(s) of a two-level or multi-level controllable experimental factor. The common practice is to use a randomized design and perform hypothesis tests on the estimates. However, such estimation and inference are not always accurate when covariate imbalance exists among the treatment groups. To overcome this issue, we propose a discrepancy-based criterion and show that the design minimizing this criterion significantly improves the accuracy of the treatment effect(s) estimates. The discrepancy-based criterion is model-free and thus makes the estimation of the treatment effect(s) robust to the model assumptions. More importantly, the proposed design is applicable to both continuous and categorical response measurements. We develop two efficient algorithms to construct the designs by optimizing the criterion for both offline and online A/B tests. Through simulation study and a real example, we show that the proposed design approach achieves good covariate balance and accurate estimation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/12/2022

Sample Constrained Treatment Effect Estimation

Treatment effect estimation is a fundamental problem in causal inference...
research
08/12/2020

Covariate Balancing Based on Kernel Density Estimates for Controlled Experiments

Controlled experiments are widely used in many applications to investiga...
research
05/19/2022

Balanced and robust randomized treatment assignments: the Finite Selection Model

The Finite Selection Model (FSM) was proposed and developed by Carl Morr...
research
01/30/2022

Stratifying Online Field Experiments Using the Pigeonhole Design

Practitioners and academics have long appreciated the benefits that expe...
research
11/03/2022

Privacy Aware Experiments without Cookies

Consider two brands that want to jointly test alternate web experiences ...
research
12/06/2020

Better Experimental Design by Hybridizing Binary Matching with Imbalance Optimization

We present a new experimental design procedure that divides a set of exp...
research
08/14/2020

Optimal Design for A/B Testing in the Presence of Covariates and Network Connection

A/B testing, also known as controlled experiments, refers to the statist...

Please sign up or login with your details

Forgot password? Click here to reset