Estimating Entropy of Distributions in Constant Space

11/18/2019
by   Jayadev Acharya, et al.
0

We consider the task of estimating the entropy of k-ary distributions from samples in the streaming model, where space is limited. Our main contribution is an algorithm that requires O(k log (1/ε)^2/ε^3) samples and a constant O(1) memory words of space and outputs a ±ε estimate of H(p). Without space limitations, the sample complexity has been established as S(k,ε)=Θ(k/εlog k+log^2 k/ε^2), which is sub-linear in the domain size k, and the current algorithms that achieve optimal sample complexity also require nearly-linear space in k. Our algorithm partitions [0,1] into intervals and estimates the entropy contribution of probability values in each interval. The intervals are designed to trade off the bias and variance of these estimates.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/19/2022

Estimation of Entropy in Constant Space with Improved Sample Complexity

Recent work of Acharya et al. (NeurIPS 2019) showed how to estimate the ...
research
02/21/2017

Stochastic Canonical Correlation Analysis

We tightly analyze the sample complexity of CCA, provide a learning algo...
research
02/22/2018

Entropy Rate Estimation for Markov Chains with Large State Space

Estimating the entropy based on data is one of the prototypical problems...
research
02/25/2021

Computing Accurate Probabilistic Estimates of One-D Entropy from Equiprobable Random Samples

We develop a simple Quantile Spacing (QS) method for accurate probabilis...
research
05/12/2023

Private and Communication-Efficient Algorithms for Entropy Estimation

Modern statistical estimation is often performed in a distributed settin...
research
05/11/2021

Targeting Makes Sample Efficiency in Auction Design

This paper introduces the targeted sampling model in optimal auction des...
research
12/12/2021

Optimal Partitions for Nonparametric Multivariate Entropy Estimation

Efficient and accurate estimation of multivariate empirical probability ...

Please sign up or login with your details

Forgot password? Click here to reset