The Holdout Randomization Test: Principled and Easy Black Box Feature Selection

11/01/2018
by   Wesley Tansey, et al.
0

We consider the problem of feature selection using black box predictive models. For example, high-throughput devices in science are routinely used to gather thousands of features for each sample in an experiment. The scientist must then sift through the many candidate features to find explanatory signals in the data, such as which genes are associated with sensitivity to a prospective therapy. Often, predictive models are used for this task: the model is fit, error on held out data is measured, and strong performing models are assumed to have discovered some fundamental properties of the system. A model-specific heuristic is then used to inspect the model parameters and rank important features, with top features reported as "discoveries." However, such heuristics provide no statistical guarantees and can produce unreliable results. We propose the holdout randomization test (HRT) as a principled approach to feature selection using black box predictive models. The HRT is similar to a permutation test, where each random reshuffling is a draw from the complete conditional distribution of the feature being tested. The HRT is model agnostic and produces a valid p-value for each feature, enabling control over the false discovery rate (or Type I error) for any predictive model. Further, the HRT is computationally efficient and, in simulations, has greater power than a competing knockoffs-based approach. Code is available at https://github.com/tansey/hrt.

READ FULL TEXT
research
10/20/2021

PPFS: Predictive Permutation Feature Selection

We propose Predictive Permutation Feature Selection (PPFS), a novel wrap...
research
06/08/2018

Black Box FDR

Analyzing large-scale, multi-experiment studies requires scientists to t...
research
02/23/2016

Auditing Black-box Models for Indirect Influence

Data-trained predictive models see widespread use, but for the most part...
research
01/28/2019

Inference after black box selection

We consider the problem of inference for parameters selected to report o...
research
10/12/2020

Embedded methods for feature selection in neural networks

The representational capacity of modern neural network architectures has...
research
10/31/2019

Sobolev Independence Criterion

We propose the Sobolev Independence Criterion (SIC), an interpretable de...
research
04/16/2021

Explorative Data Analysis of Time Series based AlgorithmFeatures of CMA-ES Variants

In this study, we analyze behaviours of the well-known CMA-ES by extract...

Please sign up or login with your details

Forgot password? Click here to reset