DeepAI AI Chat
Log In Sign Up

Designing Experiments to Measure Incrementality on Facebook

by   C. H. Bryan Liu, et al.

The importance of Facebook advertising has risen dramatically in recent years, with the platform accounting for almost 20 spend in 2017. An important consideration in advertising is incrementality: how much of the change in an experimental metric is an advertising campaign responsible for. To measure incrementality, Facebook provide lift studies. As Facebook lift studies differ from standard A/B tests, the online experimentation literature does not describe how to calculate parameters such as power and minimum sample size. Facebook also offer multi-cell lift tests, which can be used to compare campaigns that don't have statistically identical audiences. In this case, there is no literature describing how to measure the significance of the difference in incrementality between cells, or how to estimate the power or minimum sample size. We fill these gaps in the literature by providing the statistical power and required sample size calculation for Facebook lift studies. We then generalise the statistical significance, power, and required sample size calculation to multi-cell lift studies. We represent our results theoretically in terms of the distributions of test metrics and in practical terms relating to the metrics used by practitioners, making all of our code publicly available.


page 1

page 2

page 3

page 4


The non-significance factor is a simple posterior estimate of the minimum necessary sample size

A researcher is interested in what sample size is needed to get the requ...

All about sample-size calculations for A/B testing: Novel extensions and practical guide

While there exists a large amount of literature on the general challenge...

Higher significance with smaller samples: A modified Sequential Probability Ratio Test

We describe a modified sequential probability ratio test that can be use...

Large-scale analysis of user exposure to online advertising in Facebook

Online advertising is the major source of income for a large portion of ...

Detecting the skewness of data from the sample size and the five-number summary

For clinical studies with continuous outcomes, when the data are potenti...

A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms

Consistently checking the statistical significance of experimental resul...