Using Ego-Clusters to Measure Network Effects at LinkedIn

03/20/2019
by   Guillaume Saint-Jacques, et al.
0

A network effect is said to take place when a new feature not only impacts the people who receive it, but also other users of the platform, like their connections or the people who follow them. This very common phenomenon violates the fundamental assumption underpinning nearly all enterprise experimentation systems, the stable unit treatment value assumption (SUTVA). When this assumption is broken, a typical experimentation platform, which relies on Bernoulli randomization for assignment and two-sample t-test for assessment of significance, will not only fail to account for the network effect, but potentially give highly biased results. This paper outlines a simple and scalable solution to measuring network effects, using ego-network randomization, where a cluster is comprised of an "ego" (a focal individual), and her "alters" (the individuals she is immediately connected to). Our approach aims at maintaining representativity of clusters, avoiding strong modeling assumption, and significantly increasing power compared to traditional cluster-based randomization. In particular, it does not require product-specific experiment design, or high levels of investment from engineering teams, and does not require any changes to experimentation and analysis platforms, as it only requires assigning treatment an individual level. Each user either has the feature or does not, and no complex manipulation of interactions between users is needed. It focuses on measuring the one-out network effect (i.e the effect of my immediate connection's treatment on me), and gives reasonable estimates at a very low setup cost, allowing us to run such experiments dozens of times a year.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/15/2020

Minimizing Interference and Selection Bias in Network Experiment Design

Current approaches to A/B testing in networks focus on limiting interfer...
research
03/07/2018

Placebo inference on treatment effects when the number of clusters is small

I introduce a general, Fisher-style randomization testing framework to c...
research
08/11/2023

Improving Ego-Cluster for Network Effect Measurement

Network effect is common in social network platforms. Many new features ...
research
01/28/2021

Experimentation for Homogenous Policy Change

When the Stable Unit Treatment Value Assumption (SUTVA) is violated and ...
research
08/19/2020

Cluster-Adaptive Network A/B Testing: From Randomization to Estimation

A/B testing is an important decision-making tool in product development ...
research
03/20/2019

A Method for Measuring Network Effects of One-to-One Communication Features in Online A/B Tests

A/B testing is an important decision making tool in product development ...
research
08/16/2022

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

eBay's experimentation platform runs hundreds of A/B tests on any given ...

Please sign up or login with your details

Forgot password? Click here to reset