Semi-supervised multiple testing

by   David Mary, et al.

An important limitation of standard multiple testing procedures is that the null distribution should be known. Here, we consider a null distribution-free approach for multiple testing in the following semi-supervised setting: the user does not know the null distribution, but has at hand a single sample drawn from this null distribution. In practical situations, this null training sample (NTS) can come from previous experiments, from a part of the data under test, from specific simulations, or from a sampling process. In this work, we present theoretical results that handle such a framework, with a focus on the false discovery rate (FDR) control and the Benjamini-Hochberg (BH) procedure. First, we introduce a procedure providing strong FDR control. Second, we also give a power analysis for that procedure suggesting that the price to pay for ignoring the null distribution is low when the NTS sample size n is sufficiently large in front of the number of test m; namely n≳ m/(max(1,k)), where k denotes the number of "detectable" alternatives. Third, to complete the picture, we also present a negative result that evidences an intrinsic transition phase to the general semi-supervised multiple testing problem and shows that the proposed method is optimal in the sense that its performance boundary follows this transition phase. Our theoretical properties are supported by numerical experiments, which also show that the delineated boundary is of correct order without further tuning any constant. Finally, we demonstrate that our approach provides a theoretical ground for standard practice in astronomical data analysis, and in particular for the procedure proposed in <cit.> for galaxy detection.


page 1

page 2

page 3

page 4


On using empirical null distributions in Benjamini-Hochberg procedure

When performing multiple testing, adjusting the distribution of the null...

A Cramér-von Mises test of uniformity on the hypersphere

Testing uniformity of a sample supported on the hypersphere is one of th...

A direct approach to false discovery rates by decoy permutations

The current approaches to false discovery rates (FDRs) in multiple hypot...

On using empirical null distribution in Benjamini-Hochberg procedure

When performing multiple testing, adjusting the distribution of the null...

Multiple competition based FDR control

Competition based FDR control has been commonly used for over a decade i...

Optimality of the max test for detecting sparse signals with Gaussian or heavier tail

A fundamental problem in high-dimensional testing is that of global null...

Discovering the Network Granger Causality in Large Vector Autoregressive Models

This paper proposes novel inferential procedures for the network Granger...

Please sign up or login with your details

Forgot password? Click here to reset