Shrinkage Estimators in Online Experiments

04/29/2019
by   Drew Dimmery, et al.
0

We develop and analyze empirical Bayes Stein-type estimators for use in the estimation of causal effects in large-scale online experiments. While online experiments are generally thought to be distinguished by their large sample size, we focus on the multiplicity of treatment groups. The typical analysis practice is to use simple differences-in-means (perhaps with covariate adjustment) as if all treatment arms were independent. In this work we develop consistent, small bias, shrinkage estimators for this setting. In addition to achieving lower mean squared error these estimators retain important frequentist properties such as coverage under most reasonable scenarios. Modern sequential methods of experimentation and optimization such as multi-armed bandit optimization (where treatment allocations adapt over time to prior responses) benefit from the use of our shrinkage estimators. Exploration under empirical Bayes focuses more efficiently on near-optimal arms, improving the resulting decisions made under uncertainty. We demonstrate these properties by examining seventeen large-scale experiments conducted on Facebook from April to June 2017.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/24/2023

Optimal Shrinkage Estimation of Fixed Effects in Linear Panel Data Models

Shrinkage methods are frequently used to estimate fixed effects to reduc...
research
01/11/2020

Causal Inferences in Small Area Estimation

When doing impact evaluation and making causal inferences, it is importa...
research
01/27/2020

Bayesian Shrinkage Estimation of Negative Multinomial Parameter Vectors

The negative multinomial distribution is a multivariate generalization o...
research
09/13/2023

Empirical Bayes Double Shrinkage for Combining Biased and Unbiased Causal Estimates

Motivated by the proliferation of observational datasets and the need to...
research
07/24/2018

Improving pairwise comparison models using Empirical Bayes shrinkage

Comparison data arises in many important contexts, e.g. shopping, web cl...
research
05/01/2023

Leveraging covariate adjustments at scale in online A/B testing

Companies offering web services routinely run randomized online experime...
research
05/02/2022

Material Facts Obscured in Hansen's Modern Gauss-Markov Theorem

We show that the abstract and conclusion of Hansen's Econometrica paper,...

Please sign up or login with your details

Forgot password? Click here to reset