Anytime-Valid F-Tests for Faster Sequential Experimentation Through Covariate Adjustment

by   Michael Lindon, et al.

We introduce sequential F-tests and confidence sequences for subsets of coefficients of a linear model. This generalizes standard univariate Gaussian confidence sequences that are often used to perform sequential A/B tests. When performing inference on treatment effects, the ability to include covariate and treatment-covariate interaction terms reduces the stopping time of the sequential test and the width of the confidence sequences. Our sequential F-tests also have other practical applications concerning sequential tests of treatment effect heterogeneity and model selection. Our approach is based on a mixture martingale, using a Gaussian mixture over the coefficients of interest and the right-Haar mixture over the remaining model parameters. This exploits group invariance properties of the linear model to ensure that our procedure possesses a time-uniform Type I error probability of α and time-uniform 1-α coverage for all values of the nuisance parameters. This allows experiments to be continuously monitored and stopped using data dependant stopping rules. While our contributions are to provide anytime-valid guarantees, that is, time-uniform frequentist guarantees in terms of Type I error and coverage, our approach is motivated through a Bayesian framework. More specifically, our test statistic can be interpreted as a Bayes factor, which can be readily used in a traditional Bayesian analysis. Our contributions can also be viewed as providing frequentist guarantees to the Bayesian sequential test.


page 1

page 2

page 3

page 4


Doubly Robust Uniform Confidence Bands for Group-Time Conditional Average Treatment Effects in Difference-in-Differences

This study considers a panel data analysis to examine the heterogeneity ...

Anytime-valid Confidence Intervals for Contingency Tables and Beyond

E variables are tools for designing tests that keep their type-I error g...

The Generalized Oaxaca-Blinder Estimator

After performing a randomized experiment, researchers often use ordinary...

Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations

Sequential testing, always-valid p-values, and confidence sequences prom...

Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform

A/B tests are the gold standard for evaluating digital experiences on th...

Safe Sequential Testing and Effect Estimation in Stratified Count Data

Sequential decision making significantly speeds up research and is more ...

Sequential Estimation of Convex Divergences using Reverse Submartingales and Exchangeable Filtrations

We present a unified technique for sequential estimation of convex diver...

Please sign up or login with your details

Forgot password? Click here to reset