Optimal estimation of high-order missing masses, and the rare-type match problem

06/26/2023

∙

by   Stefano Favaro, et al.

∙

0

∙

share

Consider a random sample (X_1,…,X_n) from an unknown discrete distribution P=∑_j≥1p_jδ_s_j on a countable alphabet 𝕊, and let (Y_n,j)_j≥1 be the empirical frequencies of distinct symbols s_j's in the sample. We consider the problem of estimating the r-order missing mass, which is a discrete functional of P defined as θ_r(P;𝐗_n)=∑_j≥1p^r_jI(Y_n,j=0). This is generalization of the missing mass whose estimation is a classical problem in statistics, being the subject of numerous studies both in theory and methods. First, we introduce a nonparametric estimator of θ_r(P;𝐗_n) and a corresponding non-asymptotic confidence interval through concentration properties of θ_r(P;𝐗_n). Then, we investigate minimax estimation of θ_r(P;𝐗_n), which is the main contribution of our work. We show that minimax estimation is not feasible over the class of all discrete distributions on 𝕊, and not even for distributions with regularly varying tails, which only guarantee that our estimator is consistent for θ_r(P;𝐗_n). This leads to introduce the stronger assumption of second-order regular variation for the tail behaviour of P, which is proved to be sufficient for minimax estimation of θ_r(P;𝐗_n), making the proposed estimator an optimal minimax estimator of θ_r(P;𝐗_n). Our interest in the r-order missing mass arises from forensic statistics, where the estimation of the 2-order missing mass appears in connection to the estimation of the likelihood ratio T(P,𝐗_n)=θ_1(P;𝐗_n)/θ_2(P;𝐗_n), known as the "fundamental problem of forensic mathematics". We present theoretical guarantees to nonparametric estimation of T(P,𝐗_n).