DeepAI AI Chat
Log In Sign Up

Identification of unknown parameters and prediction with hierarchical matrices

04/14/2021
by   Alexander Litvinenko, et al.
0

Statistical analysis of massive datasets very often implies expensive linear algebra operations with large dense matrices. Typical tasks are an estimation of unknown parameters of the underlying statistical model and prediction of missing values. We developed the H-MLE procedure, which solves these tasks. The unknown parameters can be estimated by maximizing the joint Gaussian log-likelihood function, which depends on a covariance matrix. To decrease high computational cost, we approximate the covariance matrix in the hierarchical (H-) matrix format. The H-matrix technique allows us to work with inhomogeneous covariance matrices and almost arbitrary locations. Especially, H-matrices can be applied in cases when the matrices under consideration are dense and unstructured. For validation purposes, we implemented three machine learning methods: the k-nearest neighbors (kNN), random forest, and deep neural network. The best results (for the given datasets) were obtained by the kNN method with three or seven neighbors depending on the dataset. The results computed with the H-MLE method were compared with the results obtained by the kNN method. The developed H-matrix code and all datasets are freely available online.

READ FULL TEXT
09/08/2017

Likelihood Approximation With Hierarchical Matrices For Large Spatial Datasets

We use available measurements to estimate the unknown parameters (varian...
03/17/2023

Scalable Physics-based Maximum Likelihood Estimation using Hierarchical Matrices

Physics-based covariance models provide a systematic way to construct co...
04/24/2018

Tile Low-Rank Approximation of Large-Scale Maximum Likelihood Estimation on Manycore Architectures

Maximum likelihood estimation is an important statistical technique for ...
11/20/2018

Parallel Matrix Condensation for Calculating Log-Determinant of Large Matrix

Calculating the log-determinant of a matrix is useful for statistical co...
04/17/2017

Bayesian Hybrid Matrix Factorisation for Data Integration

We introduce a novel Bayesian hybrid matrix factorisation model (HMF) fo...