Noise Covariance Estimation in Multi-Task High-dimensional Linear Models

by   Kai Tan, et al.

This paper studies the multi-task high-dimensional linear regression models where the noise among different tasks is correlated, in the moderately high dimensional regime where sample size n and dimension p are of the same order. Our goal is to estimate the covariance matrix of the noise random vectors, or equivalently the correlation of the noise variables on any pair of two tasks. Treating the regression coefficients as a nuisance parameter, we leverage the multi-task elastic-net and multi-task lasso estimators to estimate the nuisance. By precisely understanding the bias of the squared residual matrix and by correcting this bias, we develop a novel estimator of the noise covariance that converges in Frobenius norm at the rate n^-1/2 when the covariates are Gaussian. This novel estimator is efficiently computable. Under suitable conditions, the proposed estimator of the noise covariance attains the same rate of convergence as the "oracle" estimator that knows in advance the regression coefficients of the multi-task model. The Frobenius error bounds obtained in this paper also illustrate the advantage of this new estimator compared to a method-of-moments estimator that does not attempt to estimate the nuisance. As a byproduct of our techniques, we obtain an estimate of the generalization error of the multi-task elastic-net and multi-task lasso estimators. Extensive simulation studies are carried out to illustrate the numerical performance of the proposed method.


page 14

page 15


Benign Overfitting of Non-Sparse High-Dimensional Linear Regression with Correlated Noise

We investigate the high-dimensional linear regression problem in situati...

Canonical thresholding for non-sparse high-dimensional linear regression

We consider a high-dimensional linear regression problem. Unlike many pa...

Generalized Matrix Decomposition Regression: Estimation and Inference for Two-way Structured Data

This paper studies high-dimensional regression with two-way structured d...

Multi-task Learning with High-Dimensional Noisy Images

Recent medical imaging studies have given rise to distinct but inter-rel...

Multi-task Learning for Gaussian Graphical Regressions with High Dimensional Covariates

Gaussian graphical regression is a powerful means that regresses the pre...

Sharp Bias-variance Tradeoffs of Hard Parameter Sharing in High-dimensional Linear Regression

Hard parameter sharing for multi-task learning is widely used in empiric...

Please sign up or login with your details

Forgot password? Click here to reset