Optimal Covariance Estimation for Condition Number Loss in the Spiked Model

by   David L. Donoho, et al.

We study estimation of the covariance matrix under relative condition number loss κ(Σ^-1/2Σ̂Σ^-1/2), where κ(Δ) is the condition number of matrix Δ, and Σ̂ and Σ are the estimated and theoretical covariance matrices. Optimality in κ-loss provides optimal guarantees in two stylized applications: Multi-User Covariance Estimation and Multi-Task Linear Discriminant Analysis. We assume the so-called spiked covariance model for Σ, and exploit recent advances in understanding that model, to derive a nonlinear shrinker which is asymptotically optimal among orthogonally-equivariant procedures. In our asymptotic study, the number of variables p is comparable to the number of observations n. The form of the optimal nonlinearity depends on the aspect ratio γ=p/n of the data matrix and on the top eigenvalue of Σ. For γ > 0.618..., even dependence on the top eigenvalue can be avoided. The optimal shrinker has two notable properties. First, when p/n →γ≫ 1 is large, it shrinks even very large eigenvalues substantially, by a factor 1/(1+γ). Second, even for moderate γ, certain highly statistically significant eigencomponents will be completely suppressed. We show that when γ≫ 1 is large, purely diagonal covariance matrices can be optimal, despite the top eigenvalues being large and the empirical eigenvalues being highly statistically significant. This aligns with practitioner experience. We identify intuitively reasonable procedures with small worst-case relative regret - the simplest being generalized soft thresholding having threshold at the bulk edge and slope (1+γ)^-1 above the bulk. For γ < 2 it has at most a few percent relative regret.


page 1

page 2

page 3

page 4


Extremal eigenvalues of sample covariance matrices with general population

We analyze the behavior of the largest eigenvalues of sample covariance ...

Estimation of the number of spiked eigenvalues in a covariance matrix by bulk eigenvalue matching analysis

The spiked covariance model has gained increasing popularity in high-dim...

Improving the condition number of estimated covariance matrices

High dimensional error covariance matrices are used to weight the contri...

Cleaning the covariance matrix of strongly nonstationary systems with time-independent eigenvalues

We propose a data-driven way to clean covariance matrices in strongly no...

On the Eigenstructure of Covariance Matrices with Divergent Spikes

For a generalization of Johnstone's spiked model, a covariance matrix wi...

Optimal Eigenvalue Shrinkage in the Semicircle Limit

Recent studies of high-dimensional covariance estimation often assume th...

Approximate Supermodularity Bounds for Experimental Design

This work provides performance guarantees for the greedy solution of exp...

Please sign up or login with your details

Forgot password? Click here to reset