How to Deal With Ratio Metrics When Accounting for Intra-User Correlation in A/B Testing

11/08/2019
by   Keyu Nie, et al.
0

We consider the A/B testing problem at the presence of correlation among observations coming from the same user. Furthermore, users may come from various segments where levels of correlation are different. A uniformly minimum-variance unbiased estimator of the population mean, called correlation-adjusted mean, is proposed to account for such correlation structure. It is proved theoretically and numerically better than the usual naive mean estimator and normalized mean estimator (taking average within users and then across users). The correlation-adjusted mean method is still unbiased but has reduced variance so it gains additional power. Several simulation studies are designed to show the estimation accuracy of the correlation structure, effectiveness in reducing variance, and capability of obtaining more power. An application to the eBay data is conducted to conclude this paper.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/22/2023

On Estimating the Selected Treatment Mean under a Two-Stage Adaptive Design

Adaptive designs are commonly used in clinical and drug development stud...
research
10/13/2019

Empirical and Constrained Empirical Bayes Variance Estimation Under A One Unit Per Stratum Sample Design

A single primary sampling unit (PSU) per stratum design is a popular des...
research
10/19/2021

A Bayesian Approach for the Variance of Fine Stratification

Fine stratification is a popular design as it permits the stratification...
research
08/20/2021

Parameters not identifiable or distinguishable from data, including correlation between Gaussian observations

It is shown that some theoretically identifiable parameters cannot be id...
research
05/01/2023

An unbiased non-parametric correlation estimator in the presence of ties

An inner-product Hilbert space formulation of the Kemeny distance is def...
research
03/27/2023

Adjusted Wasserstein Distributionally Robust Estimator in Statistical Learning

We propose an adjusted Wasserstein distributionally robust estimator – b...
research
02/23/2020

On a complete and sufficient statistic for the correlated Bernoulli random graph model

Inference on vertex-aligned graphs is of wide theoretical and practical ...

Please sign up or login with your details

Forgot password? Click here to reset