Agnostic tests can control the type I and type II errors simultaneously

05/11/2018
by   Victor Coscrato, et al.
1

Despite its common practice, statistical hypothesis testing presents challenges in interpretation. For instance, in the standard frequentist framework there is no control of the type II error. As a result, the non-rejection of the null hypothesis cannot reasonably be interpreted as its acceptance. We propose that this dilemma can be overcome by using agnostic hypothesis tests, since they can control the type I and II errors simultaneously. In order to make this idea operational, we show how to obtain agnostic hypothesis in typical models. For instance, we show how to build (unbiased) uniformly most powerful agnostic tests and how to obtain agnostic tests from standard p-values. Also, we present conditions such that the above tests can be made logically coherent. Finally, we present examples of consistent agnostic hypothesis tests.

READ FULL TEXT
research
04/13/2019

Interpretable hypothesis tests

Although hypothesis tests play a prominent role in Science, their interp...
research
05/11/2022

Second-Order Asymptotics of Hoeffding-Like Hypothesis Tests

We consider a binary statistical hypothesis testing problem, where n ind...
research
06/19/2020

Optimal Statistical Hypothesis Testing for Social Choice

We address the following question in this paper: "What are the most robu...
research
05/08/2018

Seeking evidence of absence: Reconsidering tests of model assumptions

Statistical tests can only reject the null hypothesis, never prove it. H...
research
12/21/2022

Powerful Partial Conjunction Hypothesis Testing via Conditioning

Research questions across a diverse array of fields are formulated as a ...
research
04/04/2022

Logical coherence in Bayesian simultaneous three-way hypothesis tests

This paper studies whether Bayesian simultaneous three-way hypothesis te...
research
03/08/2021

Exact Distribution-Free Hypothesis Tests for the Regression Function of Binary Classification via Conditional Kernel Mean Embeddings

In this paper we suggest two statistical hypothesis tests for the regres...

Please sign up or login with your details

Forgot password? Click here to reset