DeepAI AI Chat
Log In Sign Up

An evaluation framework for personalization strategy experiment designs

by   C. H. Bryan Liu, et al.

Online Controlled Experiments (OCEs) are the gold standard in evaluating the effectiveness of changes to websites. An important type of OCE evaluates different personalization strategies, which present challenges in low test power and lack of full control in group assignment. We argue that getting the right experiment setup – the allocation of users to treatment/analysis groups – should take precedence of post-hoc variance reduction techniques in order to enable the scaling of the number of experiments. We present an evaluation framework that, along with a few rule of thumbs, allow experimenters to quickly compare which experiment setup will lead to the highest probability of detecting a treatment effect under their particular circumstance.


page 1

page 2

page 3

page 4


Bayesian causal inference in automotive software engineering and online evaluation

Randomised field experiments, such as A/B testing, have long been the go...

False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments

Online controlled experiments (a.k.a. A/B testing) have been used as the...

Treatment Effect Estimation with Unmeasured Confounders in Data Fusion

In the presence of unmeasured confounders, we address the problem of tre...

Assign Experiment Variants at Scale in Online Controlled Experiments

Online controlled experiments (A/B tests) have become the gold standard ...

Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments

Online controlled experiments (also known as A/B Testing) have been view...

Uplift Modeling for Multiple Treatments with Cost Optimization

Uplift modeling is an emerging machine learning approach for estimating ...

Bias Variance Tradeoff in Analysis of Online Controlled Experiments

Many organizations utilize large-scale online controlled experiments (OC...