Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations

12/29/2022
by   Aurélien Bibaut, et al.
0

Sequential testing, always-valid p-values, and confidence sequences promise flexible statistical inference and on-the-fly decision making. However, unlike fixed-n inference based on asymptotic normality, existing sequential tests either make parametric assumptions and end up under-covering/over-rejecting when these fail or use non-parametric but conservative concentration inequalities and end up over-covering/under-rejecting. To circumvent these issues, we sidestep exact at-least-α coverage and focus on asymptotically exact coverage and asymptotic optimality. That is, we seek sequential tests whose probability of ever rejecting a true hypothesis asymptotically approaches α and whose expected time to reject a false hypothesis approaches a lower bound on all tests with asymptotic coverage at least α, both under an appropriate asymptotic regime. We permit observations to be both non-parametric and dependent and focus on testing whether the observations form a martingale difference sequence. We propose the universal sequential probability ratio test (uSPRT), a slight modification to the normal-mixture sequential probability ratio test, where we add a burn-in period and adjust thresholds accordingly. We show that even in this very general setting, the uSPRT is asymptotically optimal under mild generic conditions. We apply the results to stabilized estimating equations to test means, treatment effects, etc. Our results also provide corresponding guarantees for the implied confidence sequences. Numerical simulations verify our guarantees and the benefits of the uSPRT over alternatives.

READ FULL TEXT
research
04/30/2023

Optimal tests following sequential experiments

Recent years have seen tremendous advances in the theory and application...
research
01/04/2018

Testing Optimality of Sequential Decision-Making

This paper provides a statistical method to test whether a system that p...
research
06/27/2023

A non-parametric approach to detect patterns in binary sequences

To determine any pattern in an ordered binary sequence of wins and losse...
research
10/16/2022

Anytime-Valid F-Tests for Faster Sequential Experimentation Through Covariate Adjustment

We introduce sequential F-tests and confidence sequences for subsets of ...
research
06/09/2020

Near-Optimal Confidence Sequences for Bounded Random Variables

Many inference problems, such as sequential decision problems like A/B t...
research
02/18/2021

Off-policy Confidence Sequences

We develop confidence bounds that hold uniformly over time for off-polic...
research
10/16/2020

Nonparametric iterated-logarithm extensions of the sequential generalized likelihood ratio test

We develop a nonparametric extension of the sequential generalized likel...

Please sign up or login with your details

Forgot password? Click here to reset