A Simple Way to Deal with Cherry-picking

10/11/2018
by   Junpei Komiyama, et al.
0

Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study such selection bias in machine learning models where the reporter is motivated to promote an algorithmic innovation. When the number of possible configurations (e.g., datasets) is large, we show that the reporter can falsely report an innovation even if there is no improvement at all. We propose a `post-reporting' solution to this issue where the bias of the reported results is verified by another set of results. The theoretical findings are supported by experimental results with synthetic and real-world datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/19/2019

Custodes: Auditable Hypothesis Testing

We present Custodes: a new approach to solving the complex issue of prev...
research
11/02/2020

p-value peeking and estimating extrema

A pervasive issue in statistical hypothesis testing is that the reported...
research
05/21/2020

The Hypothesis of Testing: Paradoxes arising out of reported coronavirus case-counts

Many statisticians, epidemiologists, economists and data scientists have...
research
04/16/2019

Non-Stochastic Hypothesis Testing with Application to Privacy Against Hypothesis-Testing Adversary

In this paper, we consider privacy against hypothesis testing adversarie...
research
03/02/2023

Simultaneous Hypothesis Testing Using Internal Negative Controls with An Application to Proteomics

Negative control is a common technique in scientific investigations and ...
research
09/08/2008

Predictive Hypothesis Identification

While statistics focusses on hypothesis testing and on estimating (prope...
research
10/27/2019

Annual Interruption Rate as a KPI, its measurement and comparison

This article is divided into two chapters. The first chapter describes t...

Please sign up or login with your details

Forgot password? Click here to reset