DeepAI AI Chat
Log In Sign Up

A Bandit Approach to Multiple Testing with False Discovery Control

by   Kevin Jamieson, et al.
University of Washington

We propose an adaptive sampling approach for multiple testing which aims to maximize statistical power while ensuring anytime false discovery control. We consider n distributions whose means are partitioned by whether they are below or equal to a baseline (nulls), versus above the baseline (actual positives). In addition, each distribution can be sequentially and repeatedly sampled. Inspired by the multi-armed bandit literature, we provide an algorithm that takes as few samples as possible to exceed a target true positive proportion (i.e. proportion of actual positives discovered) while giving anytime control of the false discovery proportion (nulls predicted as actual positives). Our sample complexity results match known information theoretic lower bounds and through simulations we show a substantial performance improvement over uniform sampling and an adaptive elimination style algorithm. Given the simplicity of the approach, and its sample efficiency, the method has promise for wide adoption in the biological sciences, clinical testing for drug discovery, and online A/B/n testing problems.


Online control of the false discovery rate with decaying memory

In the online multiple testing problem, p-values corresponding to differ...

A New Perspective on Pool-Based Active Classification and False-Discovery Control

In many scientific settings there is a need for adaptive experimental de...

Simultaneous false discovery proportion bounds via knockoffs and closed testing

We propose new methods to obtain simultaneous false discovery proportion...

A factor-adjusted multiple testing of general alternatives

Factor-adjusted multiple testing is used for handling strong correlated ...

Adaptive Signal Inclusion With Genomic Applications

This paper addresses the challenge of efficiently capturing a high propo...

Lower bounds in multiple testing: A framework based on derandomized proxies

The large bulk of work in multiple testing has focused on specifying pro...

A Batched Multi-Armed Bandit Approach to News Headline Testing

Optimizing news headlines is important for publishers and media sites. A...