Reducing Interference Bias in Online Marketplace Pricing Experiments

04/26/2020
by   David Holtz, et al.
0

Online marketplace designers frequently run A/B tests to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect estimates obtained through Bernoulli randomized experiments are often biased due to violations of the stable unit treatment value assumption. This can be particularly problematic for experiments that impact sellers' strategic choices, affect buyers' preferences over items in their consideration set, or change buyers' consideration sets altogether. In this work, we measure and reduce bias due to interference in online marketplace experiments by using observational data to create clusters of similar listings, and then using those clusters to conduct cluster-randomized field experiments. We provide a lower bound on the magnitude of bias due to interference by conducting a meta-experiment that randomizes over two experiment designs: one Bernoulli randomized, one cluster randomized. In both meta-experiment arms, treatment sellers are subject to a different platform fee policy than control sellers, resulting in different prices for buyers. By conducting a joint analysis of the two meta-experiment arms, we find a large and statistically significant difference between the total average treatment effect estimates obtained with the two designs, and estimate that 32.60 interference bias. We also find weak evidence that the magnitude and/or direction of interference bias depends on extent to which a marketplace is supply- or demand-constrained, and analyze a second meta-experiment to highlight the difficulty of detecting interference bias when treatment interventions require intention-to-treat analysis.

READ FULL TEXT

page 20

page 23

research
09/13/2023

A Study of Symbiosis Bias in A/B Tests of Recommendation Algorithms

One assumption underlying the unbiasedness of global treatment effect es...
research
04/25/2021

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Two-sided marketplace platforms often run experiments to test the effect...
research
09/30/2021

Unbiased Experiments in Congested Networks

When developing a new networking algorithm, it is established practice t...
research
04/25/2020

Limiting Bias from Test-Control Interference in Online Marketplace Experiments

In an A/B test, the typical objective is to measure the total average tr...
research
12/18/2020

Statistical Properties of Exclusive and Non-exclusive Online Randomized Experiments using Bucket Reuse

Randomized experiments is a key part of product development in the tech ...
research
03/18/2022

Optimizing Randomized and Deterministic Saturation Designs under Interference

Randomized saturation designs are a family of designs which assign a pos...
research
07/22/2020

An evaluation framework for personalization strategy experiment designs

Online Controlled Experiments (OCEs) are the gold standard in evaluating...

Please sign up or login with your details

Forgot password? Click here to reset