Adaptive Neyman Allocation

by   Jinglong Zhao, et al.

In experimental design, Neyman allocation refers to the practice of allocating subjects into treated and control groups, potentially in unequal numbers proportional to their respective standard deviations, with the objective of minimizing the variance of the treatment effect estimator. This widely recognized approach increases statistical power in scenarios where the treated and control groups have different standard deviations, as is often the case in social experiments, clinical trials, marketing research, and online A/B testing. However, Neyman allocation cannot be implemented unless the standard deviations are known in advance. Fortunately, the multi-stage nature of the aforementioned applications allows the use of earlier stage observations to estimate the standard deviations, which further guide allocation decisions in later stages. In this paper, we introduce a competitive analysis framework to study this multi-stage experimental design problem. We propose a simple adaptive Neyman allocation algorithm, which almost matches the information-theoretic limit of conducting experiments. Using online A/B testing data from a social media site, we demonstrate the effectiveness of our adaptive Neyman allocation algorithm, highlighting its practicality even when applied with only a limited number of stages.


page 1

page 2

page 3

page 4


Optimal Design for A/B Testing in the Presence of Covariates and Network Connection

A/B testing, also known as controlled experiments, refers to the statist...

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

Suppose an online platform wants to compare a treatment and control poli...

Detecting Interference in A/B Testing with Increasing Allocation

In the past decade, the technology industry has adopted online randomize...

D-optimal Design for Network A/B Testing

A/B testing refers to the statistical procedure of conducting an experim...

Information, Privacy and Stability in Adaptive Data Analysis

Traditional statistical theory assumes that the analysis to be performed...

A Social Distancing-Based Facility Location Approach for Combating COVID-19

In this paper, we introduce and study the problem of facility location a...

Please sign up or login with your details

Forgot password? Click here to reset