Log In Sign Up

Estimating High-dimensional Covariance and Precision Matrices under General Missing Dependence

by   Seongoh Park, et al.

A sample covariance matrix S of completely observed data is the key statistic in a large variety of multivariate statistical procedures, such as structured covariance/precision matrix estimation, principal component analysis, and testing of equality of mean vectors. However, when the data are partially observed, the sample covariance matrix from the available data is biased and does not provide valid multivariate procedures. To correct the bias, a simple adjustment method called inverse probability weighting (IPW) has been used in previous research, yielding the IPW estimator. The estimator plays the role of S in the missing data context so that it can be plugged into off-the-shelf multivariate procedures. However, theoretical properties (e.g. concentration) of the IPW estimator have been only established under very simple missing structures; every variable of each sample is independently subject to missing with equal probability. We investigate the deviation of the IPW estimator when observations are partially observed under general missing dependency. We prove the optimal convergence rate O_p(√(log p / n)) of the IPW estimator based on the element-wise maximum norm. We also derive similar deviation results even when implicit assumptions (known mean and/or missing probability) are relaxed. The optimal rate is especially crucial in estimating a precision matrix, because of the "meta-theorem" that claims the rate of the IPW estimator governs that of the resulting precision matrix estimator. In the simulation study, we discuss non-positive semi-definiteness of the IPW estimator and compare the estimator with imputation methods, which are practically important.


page 1

page 2

page 3

page 4


Local Quadratic Spectral and Covariance Matrix Estimation

The problem of estimating the spectral density matrix f(w) of a multivar...

Minimax Estimation of Bandable Precision Matrices

The inverse covariance matrix provides considerable insight for understa...

Sub-Gaussian estimators of the mean of a random matrix with heavy-tailed entries

Estimation of the covariance matrix has attracted a lot of attention of ...

Semiparametric Inference for Non-monotone Missing-Not-at-Random Data: the No Self-Censoring Model

We study the identification and estimation of statistical functionals of...

Generalised Wishart Processes

We introduce a stochastic process with Wishart marginals: the generalise...

Cross-Validated Loss-Based Covariance Matrix Estimator Selection in High Dimensions

The covariance matrix plays a fundamental role in many modern explorator...

Non-parametric estimator of a multivariate madogram for missing-data and extreme value framework

The modeling of dependence between maxima is an important subject in sev...