DeepAI AI Chat
Log In Sign Up

Correcting for Interference in Experiments: A Case Study at Douyin

by   Vivek F. Farias, et al.

Interference is a ubiquitous problem in experiments conducted on two-sided content marketplaces, such as Douyin (China's analog of TikTok). In many cases, creators are the natural unit of experimentation, but creators interfere with each other through competition for viewers' limited time and attention. "Naive" estimators currently used in practice simply ignore the interference, but in doing so incur bias on the order of the treatment effect. We formalize the problem of inference in such experiments as one of policy evaluation. Off-policy estimators, while unbiased, are impractically high variance. We introduce a novel Monte-Carlo estimator, based on "Differences-in-Qs" (DQ) techniques, which achieves bias that is second-order in the treatment effect, while remaining sample-efficient to estimate. On the theoretical side, our contribution is to develop a generalized theory of Taylor expansions for policy evaluation, which extends DQ theory to all major MDP formulations. On the practical side, we implement our estimator on Douyin's experimentation platform, and in the process develop DQ into a truly "plug-and-play" estimator for interference in real-world settings: one which provides robust, low-bias, low-variance treatment effect estimates; admits computationally cheap, asymptotically exact uncertainty quantification; and reduces MSE by 99% compared to the best existing alternatives in our applications.


page 1

page 2

page 3

page 4


Markovian Interference in Experiments

We consider experiments in dynamical systems where interventions on some...

Average treatment effects in the presence of unknown interference

We investigate large-sample properties of treatment effect estimators un...

Regression Adjustments for Estimating the Global Treatment Effect in Experiments with Interference

Standard estimators of the global average treatment effect can be biased...

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Two-sided marketplace platforms often run experiments to test the effect...

Limiting Bias from Test-Control Interference in Online Marketplace Experiments

In an A/B test, the typical objective is to measure the total average tr...

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

Suppose an online platform wants to compare a treatment and control poli...

Algorithm is Experiment: Machine Learning, Market Design, and Policy Eligibility Rules

Algorithms produce a growing portion of decisions and recommendations bo...