An Online Sequential Test for Qualitative Treatment Effects

11/06/2021
by   Chengchun Shi, et al.
0

Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one. However, it is also critically important to detect qualitative treatment effects such that the new one may significantly outperform the existing one only under some specific circumstances. The aim of this paper is to develop a powerful testing procedure to efficiently detect such qualitative treatment effects. We propose a scalable online updating algorithm to implement our test procedure. It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control. We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justification of an efficient bootstrap method. Extensive empirical studies are conducted to examine the finite sample performance of our test procedure.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2020

A New Framework for Online Testing of Heterogeneous Treatment Effect

We propose a new framework for online testing of heterogeneous treatment...
research
02/05/2020

A Reinforcement Learning Framework for Time-Dependent Causal Effects Evaluation in A/B Testing

A/B testing, or online experiment is a standard business strategy to com...
research
12/20/2018

A Sequential Density-Based Empirical Likelihood Ratio Test for Treatment Effects

In health-related experiments, treatment effects can be identified using...
research
10/08/2016

A nonparametric sequential test for online randomized experiments

We propose a nonparametric sequential test that aims to address two prac...
research
09/02/2021

Online Testing of Subgroup Treatment Effects Based on Value Difference

Online A/B testing plays a critical role in the high-tech industry to gu...
research
08/19/2020

Cluster-Adaptive Network A/B Testing: From Randomization to Estimation

A/B testing is an important decision-making tool in product development ...
research
05/29/2021

A/B Testing for Recommender Systems in a Two-sided Marketplace

Two-sided marketplaces are standard business models of many online platf...

Please sign up or login with your details

Forgot password? Click here to reset