Direct Estimation of Information Divergence Using Nearest Neighbor Ratios

02/17/2017
by   Morteza Noshad, et al.
0

We propose a direct estimation method for Rényi and f-divergence measures based on a new graph theoretical interpretation. Suppose that we are given two sample sets X and Y, respectively with N and M samples, where η:=M/N is a constant value. Considering the k-nearest neighbor (k-NN) graph of Y in the joint data set (X,Y), we show that the average powered ratio of the number of X points to the number of Y points among all k-NN points is proportional to Rényi divergence of X and Y densities. A similar method can also be used to estimate f-divergence measures. We derive bias and variance rates, and show that for the class of γ-Hölder smooth functions, the estimator achieves the MSE rate of O(N^-2γ/(γ+d)). Furthermore, by using a weighted ensemble estimation technique, for density functions with continuous and bounded derivatives of up to the order d, and some extra conditions at the support set boundary, we derive an ensemble estimator that achieves the parametric MSE rate of O(1/N). Our estimators are more computationally tractable than other competing estimators, which makes them appealing in many practical applications.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/01/2018

Scalable Hash-Based Estimation of Divergence Measures

We propose a scalable divergence estimation method based on hashing. Con...
research
12/19/2010

Empirical estimation of entropy functionals with confidence

This paper introduces a class of k-nearest neighbor (k-NN) estimators ca...
research
02/26/2020

Minimax Optimal Estimation of KL Divergence for Continuous Distributions

Estimating Kullback-Leibler divergence from identical and independently ...
research
06/29/2019

Statistical estimation of the Kullback-Leibler divergence

Wide conditions are provided to guarantee asymptotic unbiasedness and L^...
research
06/23/2011

Relative Density-Ratio Estimation for Robust Distribution Comparison

Divergence estimators based on direct approximation of density-ratios wi...
research
11/18/2019

Consistent recovery threshold of hidden nearest neighbor graphs

Motivated by applications such as discovering strong ties in social netw...
research
01/27/2018

Scalable Mutual Information Estimation using Dependence Graphs

We propose a unified method for empirical non-parametric estimation of g...

Please sign up or login with your details

Forgot password? Click here to reset