Multivariate f-Divergence Estimation With Confidence

11/07/2014
by   Kevin R. Moon, et al.
0

The problem of f-divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In particular, even for those estimators whose MSE convergence rates are known, the asymptotic distributions are unknown. We establish the asymptotic normality of a recently proposed ensemble estimator of f-divergence between two distributions from a finite number of samples. This estimator has MSE convergence rate of O(1/T), is simple to implement, and performs well in high dimensions. This theory enables us to perform divergence-based inference tasks such as testing equality of pairs of distributions based on empirical samples. We experimentally validate our theoretical results and, as an illustration, use them to empirically bound the best achievable classification error.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/01/2018

Convergence Rates for Empirical Estimation of Binary Classification Bounds

Bounding the best achievable error probability for binary classification...
research
01/02/2018

A Concentration Result of Estimating Phi-Divergence using Data Dependent Partition

Estimation of the ϕ-divergence between two unknown probability distribut...
research
09/13/2016

Information Theoretic Structure Learning with Confidence

Information theoretic measures (e.g. the Kullback Liebler divergence and...
research
05/27/2019

Practical and Consistent Estimation of f-Divergences

The estimation of an f-divergence between two probability distributions ...
research
02/12/2014

Nonparametric Estimation of Renyi Divergence and Friends

We consider nonparametric estimation of L_2, Renyi-α and Tsallis-α diver...
research
03/28/2016

Generalized Exponential Concentration Inequality for Rényi Divergence Estimation

Estimating divergences in a consistent way is of great importance in man...
research
05/21/2019

Geometric Estimation of Multivariate Dependency

This paper proposes a geometric estimator of dependency between a pair o...

Please sign up or login with your details

Forgot password? Click here to reset