Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

09/24/2018
by   Jing Miao, et al.
0

In this paper, we provide a statistical testing framework to check whether a random sample splitting in a multi-dimensional space is carried out in a valid way, which could be directly applied to A/B testing and multivariate testing to ensure the online traffic split is truly random with respect to the covariates. We believe this is an important step of quality control that is missing in many real world online experiments. Here, we propose a randomized chi-square test method, compared with propensity score and distance components (DISCO) test methods, to test the hypothesis that the post-split categorical data sets have the same multi-dimensional distribution. The methods can be easily generalized to continuous data. We also propose a resampling procedure to adjust for multiplicity which in practice often has higher power than some existing method such as Holm's procedure. We try the three methods on both simulated and real data sets from Adobe Experience Cloud and show that each method has its own advantage while all of them establish promising power. To our knowledge, we are among the first ones to formulate the validity of A/B testing into a post-experiments statistical testing problem. Our methodology is non-parametric and requires minimum assumption on the data, so it can also have a wide range of application in other areas such as clinical trials, medicine, and recommendation system where random data splitting is needed.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/20/2020

SPlit: An Optimal Method for Data Splitting

In this article we propose an optimal method referred to as SPlit for sp...
research
03/19/2020

Homeostasis phenomenon in predictive inference when using a wrong learning model: a tale of random split of data into training and test sets

This note uses a conformal prediction procedure to provide further suppo...
research
02/09/2020

A New Framework for Online Testing of Heterogeneous Treatment Effect

We propose a new framework for online testing of heterogeneous treatment...
research
07/08/2020

Estimation and inference on high-dimensional individualized treatment rule in observational data using split-and-pooled de-correlated score

With the increasing adoption of electronic health records, there is an i...
research
09/04/2023

Selective inference after convex clustering with ℓ_1 penalization

Classical inference methods notoriously fail when applied to data-driven...
research
06/23/2022

A Diagnostic Approach to Assess the Quality of Data Splitting in Machine Learning

In machine learning, a routine practice is to split the data into a trai...
research
07/11/2018

Optimization over Continuous and Multi-dimensional Decisions with Observational Data

We consider the optimization of an uncertain objective over continuous a...

Please sign up or login with your details

Forgot password? Click here to reset