E-Valuating Classifier Two-Sample Tests

10/24/2022
by   Teodora Pandeva, et al.
0

We propose E-C2ST, a classifier two-sample test for high-dimensional data based on E-values. Compared to p-values-based tests, tests with E-values have finite sample guarantees for the type I error. E-C2ST combines ideas from existing work on split likelihood ratio tests and predictive independence testing. The resulting E-values incorporate information about the alternative hypothesis. We demonstrate the utility of E-C2ST on simulated and real-life data. In all experiments, we observe that when going from small to large sample sizes, as expected, E-C2ST starts with lower power compared to other methods but eventually converges towards one. Simultaneously, E-C2ST's type I error stays substantially below the chosen significance level, which is not always the case for the baseline methods. Finally, we use an MRI dataset to demonstrate that multiplying E-values from multiple independently conducted studies leads to a combined E-value that retains the finite sample type I error guarantees while increasing the power.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/16/2022

Finite Sample t-Tests for High-Dimensional Means

Size distortion can occur if an asymptotic testing procedure requiring d...
research
05/19/2021

Standard Curves for Empirical Likelihood Ratio Tests of Means

We present simulated standard curves for the calibration of empirical li...
research
11/01/2019

Exact model comparisons in the plausibility framework

Plausibility is a formalization of exact tests for parametric models and...
research
01/06/2023

Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values

Many testing problems are readily amenable to randomised tests such as t...
research
10/25/2022

Topology-Driven Goodness-of-Fit Tests in Arbitrary Dimensions

This paper adopts a tool from computational topology, the Euler characte...
research
08/03/2023

Robust Independence Tests with Finite Sample Guarantees for Synchronous Stochastic Linear Systems

The paper introduces robust independence tests with non-asymptotically g...
research
12/10/2019

Asymptotic based bootstrap approach for matched pairs with missingness in a single-arm

The issue of missing values is an arising difficulty when dealing with p...

Please sign up or login with your details

Forgot password? Click here to reset