Exact Paired-Permutation Testing for Structured Test Statistics

by   Ran Zmigrod, et al.

Significance testing – especially the paired-permutation test – has played a vital role in developing NLP systems to provide confidence that the difference in performance between two systems (i.e., the test statistic) is not due to luck. However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm. In this paper, we provide an efficient exact algorithm for the paired-permutation test for a family of structured test statistics. Our algorithm runs in 𝒪(GN (log GN )(log N )) time where N is the dataset size and G is the range of the test statistic. We found that our exact algorithm was 10x faster than the Monte Carlo approximation with 20000 samples on a common dataset.


page 1

page 2

page 3

page 4


Bootstrap p-values reduce type 1 error of the robust rank-order test of difference in medians

The robust rank-order test (Fligner and Policello, 1981) was designed as...

Testing Independence of Bivariate Censored Data using Random Walk on Restricted Permutation Graph

In this paper, we propose a procedure to test the independence of bivari...

Permutation tests of non-exchangeable null models

Generalizations to the permutation test are introduced to allow for situ...

The conditional permutation test

We propose a general new method, the conditional permutation test, for t...

Treatment effect validation via a permutation test in Stata

In this paper we describe the testing procedure for assessing the statis...

Exact Combinatorial Inference for Brain Images

The permutation test is known as the exact test procedure in statistics....

Revisiting the random shift approach for testing in spatial statistics

We consider the problem of non-parametric testing of independence of two...

Please sign up or login with your details

Forgot password? Click here to reset