Online control of the false discovery rate with decaying memory

by   Aaditya Ramdas, et al.

In the online multiple testing problem, p-values corresponding to different null hypotheses are observed one by one, and the decision of whether or not to reject the current hypothesis must be made immediately, after which the next p-value is observed. Alpha-investing algorithms to control the false discovery rate (FDR), formulated by Foster and Stine, have been generalized and applied to many settings, including quality-preserving databases in science and multiple A/B or multi-armed bandit tests for internet commerce. This paper improves the class of generalized alpha-investing algorithms (GAI) in four ways: (a) we show how to uniformly improve the power of the entire class of monotone GAI procedures by awarding more alpha-wealth for each rejection, giving a win-win resolution to a recent dilemma raised by Javanmard and Montanari, (b) we demonstrate how to incorporate prior weights to indicate domain knowledge of which hypotheses are likely to be non-null, (c) we allow for differing penalties for false discoveries to indicate that some hypotheses may be more important than others, (d) we define a new quantity called the decaying memory false discovery rate (mem-FDR) that may be more meaningful for truly temporal applications, and which alleviates problems that we describe and refer to as "piggybacking" and "alpha-death". Our GAI++ algorithms incorporate all four generalizations simultaneously, and reduce to more powerful variants of earlier algorithms when the weights and decay are all set to unity. Finally, we also describe a simple method to derive new online FDR rules based on an estimated false discovery proportion.


page 1

page 2

page 3

page 4


Online Rules for Control of False Discovery Rate and False Discovery Exceedance

Multiple hypothesis testing is a core problem in statistical inference a...

SAFFRON: an adaptive algorithm for online control of the false discovery rate

In the online false discovery rate (FDR) problem, one observes a possibl...

A unified treatment of multiple testing with prior knowledge using the p-filter

A significant literature studies ways of employing prior knowledge to im...

A Bandit Approach to Multiple Testing with False Discovery Control

We propose an adaptive sampling approach for multiple testing which aims...

On the consistency of adaptive multiple tests

Much effort has been done to control the "false discovery rate" (FDR) wh...

False Discovery Rate Computation: Illustrations and Modifications

False discovery rates (FDR) are an essential component of statistical in...

Dynamic Algorithms for Online Multiple Testing

We demonstrate new algorithms for online multiple testing that provably ...

Please sign up or login with your details

Forgot password? Click here to reset