Another look at the Lady Tasting Tea and permutation-based randomization tests

12/05/2019
by   Jesse Hemerik, et al.
0

Fisher's famous Lady Tasting Tea experiment is often referred to as the first permutation test or as an example of such a test. Permutation tests are special cases of the general group invariance test. Recently it has been emphasized that the set of permutations used within a permutation test should have a group structure, in the algebraic sense. If not, the test can be very anti-conservative. In this paper, however, we note that in the Lady Tasting Tea experiment, the type I error rate is controlled even if the set of permutations used does not correspond to a group. We explain the difference between permutation-based tests that fundamentally rely on a group structure, and permutation-based tests that do not. The latter are tests based on randomization of treatments. When using such tests, it can be useful to consider a randomization scheme that does correspond to a group. In particular, we can use randomization schemes where the number of possible treatment patterns is larger than in standard permutation-based randomization tests. This leads to exact p-values of improved resolution, providing increased power for very small significance levels. We discuss applications in clinical trials and elsewhere.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset