Optimal Randomized Approximations for Matrix based Renyi's Entropy

05/16/2022
by   Yuxin Dong, et al.
5

The Matrix-based Renyi's entropy enables us to directly measure information quantities from given data without the costly probability density estimation of underlying distributions, thus has been widely adopted in numerous statistical learning and inference tasks. However, exactly calculating this new information quantity requires access to the eigenspectrum of a semi-positive definite (SPD) matrix A which grows linearly with the number of samples n, resulting in a O(n^3) time complexity that is prohibitive for large-scale applications. To address this issue, this paper takes advantage of stochastic trace approximations for matrix-based Renyi's entropy with arbitrary α∈ R^+ orders, lowering the complexity by converting the entropy approximation to a matrix-vector multiplication problem. Specifically, we develop random approximations for integer order α cases and polynomial series approximations (Taylor and Chebyshev) for non-integer α cases, leading to a O(n^2sm) overall time complexity, where s,m ≪ n denote the number of vector queries and the polynomial order respectively. We theoretically establish statistical guarantees for all approximation algorithms and give explicit order of s and m with respect to the approximation error ε, showing optimal convergence rate for both parameters up to a logarithmic factor. Large-scale simulations and real-world applications validate the effectiveness of the developed approximations, demonstrating remarkable speedup with negligible loss in accuracy.

READ FULL TEXT

page 1

page 15

research
12/27/2021

Computationally Efficient Approximations for Matrix-based Renyi's Entropy

The recently developed matrix based Renyi's entropy enables measurement ...
research
11/30/2022

Robust and Fast Measure of Information via Low-rank Representation

The matrix-based Rényi's entropy allows us to directly quantify informat...
research
10/19/2020

Hutch++: Optimal Stochastic Trace Estimation

We study the problem of estimating the trace of a matrix A that can only...
research
11/12/2018

Fast Computing von Neumann Entropy for Large-scale Graphs via Quadratic Approximations

The von Neumann graph entropy (VNGE) can be used as a measure of graph c...
research
08/24/2022

Randomized sketching for Krylov approximations of large-scale matrix functions

The computation of f(A)b, the action of a matrix function on a vector, i...
research
07/14/2021

Scalable Optimal Transport in High Dimensions for Graph Distances, Embedding Alignment, and More

The current best practice for computing optimal transport (OT) is via en...
research
03/27/2013

Approximations in Bayesian Belief Universe for Knowledge Based Systems

When expert systems based on causal probabilistic networks (CPNs) reach ...

Please sign up or login with your details

Forgot password? Click here to reset