Anytime-Valid F-Tests for Faster Sequential Experimentation Through Covariate Adjustment

10/16/2022
by   Michael Lindon, et al.
0

We introduce sequential F-tests and confidence sequences for subsets of coefficients of a linear model. This generalizes standard univariate Gaussian confidence sequences that are often used to perform sequential A/B tests. When performing inference on treatment effects, the ability to include covariate and treatment-covariate interaction terms reduces the stopping time of the sequential test and the width of the confidence sequences. Our sequential F-tests also have other practical applications concerning sequential tests of treatment effect heterogeneity and model selection. Our approach is based on a mixture martingale, using a Gaussian mixture over the coefficients of interest and the right-Haar mixture over the remaining model parameters. This exploits group invariance properties of the linear model to ensure that our procedure possesses a time-uniform Type I error probability of α and time-uniform 1-α coverage for all values of the nuisance parameters. This allows experiments to be continuously monitored and stopped using data dependant stopping rules. While our contributions are to provide anytime-valid guarantees, that is, time-uniform frequentist guarantees in terms of Type I error and coverage, our approach is motivated through a Bayesian framework. More specifically, our test statistic can be interpreted as a Bayes factor, which can be readily used in a traditional Bayesian analysis. Our contributions can also be viewed as providing frequentist guarantees to the Bayesian sequential test.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/03/2023

Doubly Robust Uniform Confidence Bands for Group-Time Conditional Average Treatment Effects in Difference-in-Differences

This study considers a panel data analysis to examine the heterogeneity ...
research
03/18/2022

Anytime-valid Confidence Intervals for Contingency Tables and Beyond

E variables are tools for designing tests that keep their type-I error g...
research
04/24/2020

The Generalized Oaxaca-Blinder Estimator

After performing a randomized experiment, researchers often use ordinary...
research
12/29/2022

Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations

Sequential testing, always-valid p-values, and confidence sequences prom...
research
02/20/2023

Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform

A/B tests are the gold standard for evaluating digital experiences on th...
research
02/22/2023

Safe Sequential Testing and Effect Estimation in Stratified Count Data

Sequential decision making significantly speeds up research and is more ...
research
03/16/2021

Sequential Estimation of Convex Divergences using Reverse Submartingales and Exchangeable Filtrations

We present a unified technique for sequential estimation of convex diver...

Please sign up or login with your details

Forgot password? Click here to reset