A New Framework for Online Testing of Heterogeneous Treatment Effect

02/09/2020
by   Miao Yu, et al.
0

We propose a new framework for online testing of heterogeneous treatment effects. The proposed test, named sequential score test (SST), is able to control type I error under continuous monitoring and detect multi-dimensional heterogeneous treatment effects. We provide an online p-value calculation for SST, making it convenient for continuous monitoring, and extend our tests to online multiple testing settings by controlling the false discovery rate. We examine the empirical performance of the proposed tests and compare them with a state-of-art online test, named mSPRT using simulations and a real data. The results show that our proposed test controls type I error at any time, has higher detection power and allows quick inference on online A/B testing.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/02/2021

Online Testing of Subgroup Treatment Effects Based on Value Difference

Online A/B testing plays a critical role in the high-tech industry to gu...
research
11/06/2021

An Online Sequential Test for Qualitative Treatment Effects

Tech companies (e.g., Google or Facebook) often use randomized online ex...
research
06/16/2017

A framework for Multi-A(rmed)/B(andit) testing with online FDR control

We propose an alternative framework to existing setups for controlling f...
research
10/08/2016

A nonparametric sequential test for online randomized experiments

We propose a nonparametric sequential test that aims to address two prac...
research
10/14/2021

Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments

Online controlled experiments (also known as A/B Testing) have been view...
research
09/24/2018

Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

In this paper, we provide a statistical testing framework to check wheth...
research
09/13/2020

Which Wilcoxon should we use? An interactive rank test and other alternatives

Classical nonparametric tests to compare multiple samples, such as the W...

Please sign up or login with your details

Forgot password? Click here to reset