
The Structure of Optimal Private Tests for Simple Hypotheses
Hypothesis testing plays a central role in statistical inference, and is...
read it

Once is Never Enough: Foundations for Sound Statistical Inference in Tor Network Experimentation
Tor is a popular lowlatency anonymous communication system that focuses...
read it

Dataadaptive statistics for multiple hypothesis testing in highdimensional settings
Current statistical inference problems in areas like astronomy, genomics...
read it

Permutationbased true discovery guarantee by sum tests
Sumbased global tests are highly popular in multiple hypothesis testing...
read it

Global and Local TwoSample Tests via Regression
Twosample testing is a fundamental problem in statistics. Despite its l...
read it

A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms
Consistently checking the statistical significance of experimental resul...
read it

Challenges in Bayesian Adaptive Data Analysis
Traditional statistical analysis requires that the analysis process and ...
read it
A Rademacher Complexity Based Method fo rControlling Power and Confidence Level in Adaptive Statistical Analysis
While standard statistical inference techniques and machine learning generalization bounds assume that tests are run on data selected independently of the hypotheses, practical data analysis and machine learning are usually iterative and adaptive processes where the same holdout data is often used for testing a sequence of hypotheses (or models), which may each depend on the outcome of the previous tests on the same data. In this work, we present RadaBound a rigorous, efficient and practical procedure for controlling the generalization error when using a holdout sample for multiple adaptive testing. Our solution is based on a new application of the Rademacher Complexity generalization bounds, adapted to dependent tests. We demonstrate the statistical power and practicality of our method through extensive simulations and comparisons to alternative approaches.
READ FULL TEXT
Comments
There are no comments yet.