Significance testing in non-sparse high-dimensional linear models

10/07/2016
by   Yinchu Zhu, et al.
0

In high-dimensional linear models, the sparsity assumption is typically made, stating that most of the parameters are equal to zero. Under the sparsity assumption, estimation and, recently, inference have been well studied. However, in practice, sparsity assumption is not checkable and more importantly is often violated, with a large number of covariates expected to be associated with the response, indicating that possibly all, rather than just a few, parameters are non-zero. A natural example is a genome-wide gene expression profiling, where all genes are believed to affect a common disease marker. We show that existing inferential methods are sensitive to the sparsity assumption, and may, in turn, result in the severe lack of control of Type-I error. In this article, we propose a new inferential method, named CorrT, which is robust to model misspecification and adaptive to the sparsity assumption. CorrT is shown to have Type I error approaching the nominal level for any models and Type II error approaching zero for sparse and many dense models. In fact, CorrT is also shown to be optimal in a variety of frameworks: sparse, non-sparse and hybrid models where sparse and dense signals are mixed. Numerical experiments show a favorable performance of the CorrT test compared to the state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/17/2022

Simultaneous Inference in Non-Sparse High-Dimensional Linear Models

Inference and prediction under the sparsity assumption have been a hot r...
research
10/10/2016

Linear Hypothesis Testing in Dense High-Dimensional Linear Models

We propose a methodology for testing linear hypothesis in high-dimension...
research
08/01/2017

Breaking the curse of dimensionality in regression

Models with many signals, high-dimensional models, often impose structur...
research
10/14/2016

Two-sample testing in non-sparse high-dimensional linear models

In analyzing high-dimensional models, sparsity of the model parameter is...
research
02/26/2018

Testability of high-dimensional linear models with non-sparse structures

This paper studies hypothesis testing and confidence interval constructi...
research
06/06/2020

Chromatic Learning for Sparse Datasets

Learning over sparse, high-dimensional data frequently necessitates the ...
research
09/09/2019

Robust testing in generalized linear models by sign-flipping score contributions

Generalized linear models are often misspecified due to overdispersion, ...

Please sign up or login with your details

Forgot password? Click here to reset