Fair Effect Attribution in Parallel Online Experiments

10/15/2022
by   Alexander Buchholz, et al.
0

A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly in treatment and control groups. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes such as engagement metrics. These are measured globally and monitored to protect overall user experience. Therefore, it is crucial to measure these interaction effects and attribute their overall impact in a fair way to the respective experimenters. We suggest an approach to measure and disentangle the effect of simultaneous experiments by providing a cost sharing approach based on Shapley values. We also provide a counterfactual perspective, that predicts shared impact based on conditional average treatment effects making use of causal inference techniques. We illustrate our approach in real world and synthetic data experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/18/2019

Causal inference from observational data: Estimating the effect of contributions on visitation frequency atLinkedIn

Randomized experiments (A/B testings) have become the standard way for w...
research
03/16/2018

Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

Online controlled experiments are the primary tool for measuring the cau...
research
03/28/2022

Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments

Researchers are increasingly turning to machine learning (ML) algorithms...
research
09/30/2021

Unbiased Experiments in Congested Networks

When developing a new networking algorithm, it is established practice t...
research
07/14/2021

Online Evaluation Methods for the Causal Effect of Recommendations

Evaluating the causal effect of recommendations is an important objectiv...
research
11/14/2022

Partial counterfactual identification and uplift modeling: theoretical results and real-world assessment

Counterfactuals are central in causal human reasoning and the scientific...
research
02/18/2021

Novelty and Primacy: A Long-Term Estimator for Online Experiments

Online experiments are the gold standard for evaluating impact on user e...

Please sign up or login with your details

Forgot password? Click here to reset