Supplemental Studies for Simultaneous Goodness-of-Fit Testing

by   Wolfgang Rolke, et al.

Testing to see whether a given data set comes from some specified distribution is among the oldest types of problems in Statistics. Many such tests have been developed and their performance studied. The general result has been that while a certain test might perform well, aka have good power, in one situation it will fail badly in others. This is not a surprise given the great many ways in which a distribution can differ from the one specified in the null hypothesis. It is therefore very difficult to decide a priori which test to use. The obvious solution is not to rely on any one test but to run several of them. This however leads to the problem of simultaneous inference, that is, if several tests are done even if the null hypothesis were true, one of them is likely to reject it anyway just by random chance. In this paper we present a method that yields a p value that is uniform under the null hypothesis no matter how many tests are run. This is achieved by adjusting the p value via simulation. We present a number of simulation studies that show the uniformity of the p value and others that show that this test is superior to any one test if the power is averaged over a large number of cases.


page 1

page 2

page 3

page 4


P-value: A Bless or A Curse for Evidence-Based Studies?

As a convention, p-value is often computed in frequentist hypothesis tes...

Reallocating and Resampling: A Comparison for Inference

Simulation-based inference plays a major role in modern statistics, and ...

A Plot is Worth a Thousand Tests: Assessing Residual Diagnostics with the Lineup Protocol

Regression experts consistently recommend plotting residuals for model d...

A general adaptive framework for multivariate point null testing

As a common step in refining their scientific inquiry, investigators are...

Communication-Efficient Distribution-Free Inference Over Networks

Consider a star network where each local node possesses a set of distrib...

With Little Power Comes Great Responsibility

Despite its importance to experimental design, statistical power (the pr...

Logical coherence in Bayesian simultaneous three-way hypothesis tests

This paper studies whether Bayesian simultaneous three-way hypothesis te...

Please sign up or login with your details

Forgot password? Click here to reset