Improved Covariance Matrix Estimator using Shrinkage Transformation and Random Matrix Theory

12/08/2019
by   Samruddhi Deshmukh, et al.
0

One of the major challenges in multivariate analysis is the estimation of population covariance matrix from sample covariance matrix (SCM). Most recent covariance matrix estimators use either shrinkage transformations or asymptotic results from Random Matrix Theory (RMT). Shrinkage techniques help in pulling extreme correlation values towards certain target values whereas tools from RMT help in removing noisy eigenvalues of SCM. Both of these techniques use different approaches to achieve a similar goal which is to remove noisy correlations and add structure to SCM to overcome the bias-variance trade-off. In this paper, we first critically evaluate the pros and cons of these two techniques and then propose an improved estimator which exploits the advantages of both by taking an optimally weighted convex combination of covariance matrices estimated by an improved shrinkage transformation and a RMT based filter. It is a generalized estimator which can adapt to changing sampling noise conditions in various datasets by performing hyperparameter optimization. We show the effectiveness of this estimator on the problem of designing a financial portfolio with minimum risk. We have chosen this problem because the complex properties of stock market data provide extreme conditions to test the robustness of a covariance estimator. Using data from four of the world's largest stock exchanges, we show that our proposed estimator outperforms existing estimators in minimizing the out-of-sample risk of the portfolio and hence predicts population statistics more precisely. Since covariance analysis is a crucial statistical tool, this estimator can be used in a wide range of machine learning, signal processing and high dimensional pattern recognition applications.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/25/2022

An Improvement on the Hotelling T^2 Test Using the Ledoit-Wolf Nonlinear Shrinkage Estimator

Hotelling's T^2 test is a classical approach for discriminating the mean...
research
09/10/2019

Covariance Matrix Estimation under Total Positivity for Portfolio Selection

Selecting the optimal Markowitz porfolio depends on estimating the covar...
research
10/23/2017

Structural Variability from Noisy Tomographic Projections

In cryo-electron microscopy, the 3D electric potentials of an ensemble o...
research
07/03/2021

Cleaning large-dimensional covariance matrices for correlated samples

A non-linear shrinkage estimator of large-dimensional covariance matrice...
research
03/04/2015

Large Dimensional Analysis of Robust M-Estimators of Covariance with Outliers

A large dimensional characterization of robust M-estimators of covarianc...
research
11/15/2017

The Dispersion Bias

Estimation error has plagued quantitative finance since Harry Markowitz ...
research
10/27/2020

Source Enumeration via RMT Estimator Based on Linear Shrinkage Estimation of Noise Eigenvalues Using Relatively Few Samples

Estimating the number of signals embedded in noise is a fundamental prob...

Please sign up or login with your details

Forgot password? Click here to reset