The Significance Filter, the Winner's Curse and the Need to Shrink

09/20/2020
by   Erik van Zwet, et al.
0

The "significance filter" refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we condition on significance. In fact, the significance filter leads to overestimation of the magnitude of the parameter, which has been called the "winner's curse". It can also lead to undercoverage of the confidence interval. Moreover, these problems become more severe if the power is low. While these issues clearly deserve our attention, they have been studied only informally and mathematical results are lacking. Here we study them from the frequentist and the Bayesian perspective. We prove that the relative bias of the magnitude is a decreasing function of the power and that the usual confidence interval undercovers when the power is less than 50 conclude that failure to apply the appropriate amount of shrinkage can lead to misleading inferences.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/26/2021

Mathematical properties and finite-population correction for the Wilson score interval

In this paper we examine the properties of the Wilson score interval, us...
research
03/06/2023

Differentially Private Confidence Interval for Extrema of Parameters

This paper aims to construct a valid and efficient confidence interval f...
research
04/10/2019

Practical Valid Inferences for the Two-Sample Binomial Problem

Consider comparing two independent binomial responses. Our interest is w...
research
11/11/2022

Assessing the Lognormal Distribution Assumption For the Crude Odds Ratio: Implications For Point and Interval Estimation

The assumption that the sampling distribution of the crude odds ratio (O...
research
10/25/2018

Optimal post-selection inference for sparse signals: a nonparametric empirical-Bayes approach

A large body of recent Bayesian work has focused on the question of how ...
research
07/25/2017

Aspects of Chaitin's Omega

The halting probability of a Turing machine,also known as Chaitin's Omeg...

Please sign up or login with your details

Forgot password? Click here to reset