Empirical Bayes for Large-scale Randomized Experiments: a Spectral Approach
share
Large-scale randomized experiments, sometimes called A/B tests, are increasingly prevalent in many industries. Though such experiments are often analyzed via frequentist t-tests, arguably such analyses are deficient: p-values are hard to interpret and not easily incorporated into decision-making. As an alternative, we propose an empirical Bayes approach, which assumes that experiments come from a population, and therefore the treatment effects are realized from a "true prior". A key step in implementing this framework is to estimate the underlying true prior from a set of previous experiments. First, we show that the empirical effect estimates from individual experiments can be asymptotically modeled as independent draws from the true prior perturbed by additive Gaussian noise with heterogeneous scales. Second, following the work of Robbins, we generalize from estimating the prior to estimating a family of marginal densities of the empirical effect estimates, indexed by the noise scale. We show that this density family is characterized by the heat equation. Third, given the general form of solution to the heat equation, we develop a spectral maximum likelihood estimate based on a Fourier series representation, which can be efficiently computed via convex optimization. In order to select hyperparameters and compare models we describe two model selection criteria. Finally, we demonstrate our method on simulated and real data, and compare posterior inference to that under a Gaussian mixture model for the prior.
READ FULL TEXT