Percent Change Estimation in Large Scale Online Experiments

11/01/2017

∙

by   Jacopo Soriano, et al.

∙

0

∙

share

Online experiments are a fundamental component of the development of web-facing products. Given the large user-base, even small product improvements can have a large impact on an absolute scale. As a result, accurately estimating the relative impact of these changes is extremely important. I propose an approach based on an objective Bayesian model to improve the sensitivity of percent change estimation in A/B experiments. Leveraging pre-period information, this approach produces more robust and accurate point estimates and up to 50 The R package abpackage provides an implementation of the approach.