Automatic Detection and Diagnosis of Biased Online Experiments

07/31/2018
by   Nanyu Chen, et al.
0

We have seen a massive growth of online experiments at LinkedIn, and in industry at large. It is now more important than ever to create an intelligent A/B platform that can truly democratize A/B testing by allowing everyone to make quality decisions, regardless of their skillset. With the tremendous knowledge base created around experimentation, we are able to mine through historical data, and discover the most common causes for biased experiments. In this paper, we share four of such common causes, and how we build into our A/B testing platform the automatic detection and diagnosis of such root causes. These root causes range from design-imposed bias, self-selection bias, novelty effect and trigger-day effect. We will discuss in detail what each bias is and the scalable algorithm we developed to detect the bias. Surfacing up the existence and root cause of bias automatically for every experiment is an important milestone towards intelligent A/B testing.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/25/2021

Bias in Machine Learning Software: Why? How? What to do?

Increasingly, software is making autonomous decisions in case of crimina...
research
03/22/2018

The Roots of Bias on Uber

In the last decade, there has been a growth in, what we call, digitally ...
research
09/19/2018

Causal Testing: Finding Defects' Root Causes

Isolating and repairing unexpected or buggy software behavior typically ...
research
03/09/2023

RCABench: Open Benchmarking Platform for Root Cause Analysis

Fuzzing has contributed to automatically identifying bugs and vulnerabil...
research
05/27/2023

Counterfactual Formulation of Patient-Specific Root Causes of Disease

Root causes of disease intuitively correspond to root vertices that incr...
research
05/25/2023

Learning DAGs from Data with Few Root Causes

We present a novel perspective and algorithm for learning directed acycl...
research
02/06/2019

Heavy User Effect in A/B Testing: Identification and Estimation

On-line experimentation (also known as A/B testing) has become an integr...

Please sign up or login with your details

Forgot password? Click here to reset