DeepAI AI Chat
Log In Sign Up

Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

by   C. H. Bryan Liu, et al.

Online controlled experiments are the primary tool for measuring the causal impact of product changes in digital businesses. It is increasingly common for digital products and services to interact with customers in a personalised way. Using online controlled experiments to optimise personalised interaction strategies is challenging because the usual assumption of statistically equivalent user groups is violated. Additionally, challenges are introduced by users qualifying for strategies based on dynamic, stochastic attributes. Traditional A/B tests can salvage statistical equivalence by pre-allocating users to control and exposed groups, but this dilutes the experimental metrics and reduces the test power. We present a stacked incrementality test framework that addresses problems with running online experiments for personalised user strategies. We derive bounds that show that our framework is superior to the best simple A/B test given enough users and that this condition is easily met for large scale online experiments. In addition, we provide a test power calculator and describe a selection of pitfalls and lessons learnt from our experience using it.


Datasets for Online Controlled Experiments

Online Controlled Experiments (OCE) are the gold standard to measure imp...

Bias Variance Tradeoff in Analysis of Online Controlled Experiments

Many organizations utilize large-scale online controlled experiments (OC...

Covariance Estimation and its Application in Large-Scale Online Controlled Experiments

During the last few decades, online controlled experiments (also known a...

Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

The rise of internet-based services and products in the late 1990's brou...

Fair Effect Attribution in Parallel Online Experiments

A/B tests serve the purpose of reliably identifying the effect of change...

Assign Experiment Variants at Scale in Online Controlled Experiments

Online controlled experiments (A/B tests) have become the gold standard ...

Percent Change Estimation in Large Scale Online Experiments

Online experiments are a fundamental component of the development of web...