All about sample-size calculations for A/B testing: Novel extensions and practical guide

05/25/2023
by   Jing Zhou, et al.
0

While there exists a large amount of literature on the general challenges of and best practices for trustworthy online A/B testing, there are limited studies on sample size estimation, which plays a crucial role in trustworthy and efficient A/B testing that ensures the resulting inference has a sufficient power and type I error control. For example, when sample size is under-estimated, the statistical inference, even with the correct analysis methods, will not be able to detect the true significant improvement leading to misinformed and costly decisions. This paper addresses this fundamental gap by developing new sample size calculation methods for correlated data, as well as absolute vs. relative treatment effects, both ubiquitous in online experiments. Additionally, we address a practical question of the minimal observed difference that will be statistically significant and how it relates to average treatment effect and sample size calculation. All proposed methods are accompanied by mathematical proofs, illustrative examples, and simulations. We end by sharing some best practices on various practical topics on sample size calculation and experimental design.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/26/2022

Bayesian design and analysis of two-arm cluster randomised trials using assurance

We consider the design of a two-arm superiority cluster randomised contr...
research
05/12/2021

Sample size planning for pilot studies

Pilot studies are often the first step of experimental research. It is u...
research
05/20/2023

Power and sample size calculations for testing the ratio of reproductive values in phylogenetic samples

The quality of the inferences we make from pathogen sequence data is det...
research
06/07/2018

Designing Experiments to Measure Incrementality on Facebook

The importance of Facebook advertising has risen dramatically in recent ...
research
08/05/2019

Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

This work presents a statistically principled method for estimating the ...
research
10/14/2019

The need for adequate sampling in a well-functioning market surveillance system

Adequate sampling is essential for the well-functioning of a market surv...
research
06/10/2018

Sample size calculation and blinded recalculation for analysis of covariance models with multiple random covariates

When testing for superiority in a parallel-group setting with a continuo...

Please sign up or login with your details

Forgot password? Click here to reset