DeepAI AI Chat
Log In Sign Up

An evaluation framework for personalization strategy experiment designs

07/22/2020
by   C. H. Bryan Liu, et al.
0

Online Controlled Experiments (OCEs) are the gold standard in evaluating the effectiveness of changes to websites. An important type of OCE evaluates different personalization strategies, which present challenges in low test power and lack of full control in group assignment. We argue that getting the right experiment setup – the allocation of users to treatment/analysis groups – should take precedence of post-hoc variance reduction techniques in order to enable the scaling of the number of experiments. We present an evaluation framework that, along with a few rule of thumbs, allow experimenters to quickly compare which experiment setup will lead to the highest probability of detecting a treatment effect under their particular circumstance.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/01/2022

Bayesian causal inference in automotive software engineering and online evaluation

Randomised field experiments, such as A/B testing, have long been the go...
08/14/2018

False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments

Online controlled experiments (a.k.a. A/B testing) have been used as the...
08/23/2022

Treatment Effect Estimation with Unmeasured Confounders in Data Fusion

In the presence of unmeasured confounders, we address the problem of tre...
12/17/2022

Assign Experiment Variants at Scale in Online Controlled Experiments

Online controlled experiments (A/B tests) have become the gold standard ...
10/14/2021

Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments

Online controlled experiments (also known as A/B Testing) have been view...
08/14/2019

Uplift Modeling for Multiple Treatments with Cost Optimization

Uplift modeling is an emerging machine learning approach for estimating ...
09/10/2020

Bias Variance Tradeoff in Analysis of Online Controlled Experiments

Many organizations utilize large-scale online controlled experiments (OC...