Representation-Aware Experimentation: Group Inequality Analysis for A/B Testing and Alerting

by   Rina Friedberg, et al.

As companies adopt increasingly experimentation-driven cultures, it is crucial to develop methods for understanding any potential unintended consequences of those experiments. We might have specific questions about those consequences (did a change increase or decrease gender representation equality among content creators?); we might also wonder whether if we have not yet considered the right question (that is, we don't know what we don't know). Hence we address the problem of unintended consequences in experimentation from two perspectives: namely, pre-specified vs. data-driven selection, of dimensions of interest. For a specified dimension, we introduce a statistic to measure deviation from equal representation (DER statistic), give its asymptotic distribution, and evaluate finite-sample performance. We explain how to use this statistic to search across large-scale experimentation systems to alert us to any extreme unintended consequences on group representation. We complement this methodology by discussing a search for heterogeneous treatment effects along a set of dimensions with causal trees, modified slightly for practicalities in our ecosystem, and used here as a way to dive deeper into experiments flagged by the DER statistic alerts. We introduce a method for simulating data that closely mimics observed data at LinkedIn, and evaluate the performance of DER statistics in simulations. Last, we give a case study from LinkedIn, and show how these methodologies empowered us to discover surprising and important insights about group representation. Code for replication is available in an appendix.


page 1

page 2

page 3

page 4


A Small-Uniform Statistic for the Inference of Functional Linear Regressions

We propose a "small-uniform" statistic for the inference of the function...

An Extreme-Value Approach for Testing the Equality of Large U-Statistic Based Correlation Matrices

There has been an increasing interest in testing the equality of large P...

Matching Bounds: How Choice of Matching Algorithm Impacts Treatment Effects Estimates and What to Do about It

Different matches on the same data may produce different treatment effec...

Kaplan-Meier V and U-statistics

In this paper we study Kaplan-Meier V and U-statistics defined as θ(F̂_n...

Practical Policy Optimization with Personalized Experimentation

Many organizations measure treatment effects via an experimentation plat...

Kendall's Tau for Two-Sample Inference Problems

We consider a Kendall's tau measure between a binary group indicator and...

Incorporating Structural Stigma into Network Analysis

A rich literature has explored the modeling of homophily and other forms...

Please sign up or login with your details

Forgot password? Click here to reset