Optimal rates of entropy estimation over Lipschitz balls
share
We consider the problem of minimax estimation of the entropy of a density over Lipschitz balls. Dropping the usual assumption that the density is bounded away from zero, we obtain the minimax rates (n n)^-s/s+d + n^-1/2 for 0<s≤ 2 in arbitrary dimension d, where s is the smoothness parameter and n is the number of independent samples. Using a two-stage approximation technique, which first approximate the density by its kernel-smoothed version, and then approximate the non-smooth functional by polynomials, we construct entropy estimators that attain the minimax rate of convergence, shown optimal by matching lower bounds. One of the key steps in analyzing the bias relies on a novel application of the Hardy-Littlewood maximal inequality, which also leads to a new inequality on Fisher information that might be of independent interest.
READ FULL TEXT