Causal Inference Struggles with Agency on Online Platforms

07/19/2021
by   Smitha Milli, et al.
3

Online platforms regularly conduct randomized experiments to understand how changes to the platform causally affect various outcomes of interest. However, experimentation on online platforms has been criticized for having, among other issues, a lack of meaningful oversight and user consent. As platforms give users greater agency, it becomes possible to conduct observational studies in which users self-select into the treatment of interest as an alternative to experiments in which the platform controls whether the user receives treatment or not. In this paper, we conduct four large-scale within-study comparisons on Twitter aimed at assessing the effectiveness of observational studies derived from user self-selection on online platforms. In a within-study comparison, treatment effects from an observational study are assessed based on how effectively they replicate results from a randomized experiment with the same target population. We test the naive difference in group means estimator, exact matching, regression adjustment, and inverse probability of treatment weighting while controlling for plausible confounding variables. In all cases, all observational estimates perform poorly at recovering the ground-truth estimate from the analogous randomized experiments. In all cases except one, the observational estimates have the opposite sign of the randomized estimate. Our results suggest that observational studies derived from user self-selection are a poor alternative to randomized experimentation on online platforms. In discussing our results, we postulate "Catch-22"s that suggest that the success of causal inference in these settings may be at odds with the original motivations for providing users with greater agency.

READ FULL TEXT

page 2

page 4

page 13

research
11/13/2017

Causal Inference from Observational Studies with Clustered Interference

Inferring causal effects from an observational study is challenging beca...
research
03/13/2023

Observational Causal Inference in Novel Diseases: A Case Study of COVID-19

A key issue for all observational causal inference is that it relies on ...
research
07/17/2019

Assessing Treatment Effect Variation in Observational Studies: Results from a Data Challenge

A growing number of methods aim to assess the challenging question of tr...
research
11/09/2022

Strategy to select most efficient RCT samples based on observational data

Randomized experiments can provide unbiased estimates of sample average ...
research
06/09/2023

Using Auxiliary Data to Boost Precision in the Analysis of A/B Tests on an Online Educational Platform: New Data and New Results

Randomized A/B tests within online learning platforms represent an excit...
research
06/07/2022

Confounder Analysis in Measuring Representation in Product Funnels

This paper discusses an application of Shapley values in the causal infe...

Please sign up or login with your details

Forgot password? Click here to reset