Sharper Utility Bounds for Differentially Private Models
share
In this paper, by introducing Generalized Bernstein condition, we propose the first 𝒪(√(p)/nϵ) high probability excess population risk bound for differentially private algorithms under the assumptions G-Lipschitz, L-smooth, and Polyak-Łojasiewicz condition, based on gradient perturbation method. If we replace the properties G-Lipschitz and L-smooth by α-Hölder smoothness (which can be used in non-smooth setting), the high probability bound comes to 𝒪(n^-α/1+2α) w.r.t n, which cannot achieve 𝒪(1/n) when α∈(0,1]. To solve this problem, we propose a variant of gradient perturbation method, max{1,g}-Normalized Gradient Perturbation (m-NGP). We further show that by normalization, the high probability excess population risk bound under assumptions α-Hölder smooth and Polyak-Łojasiewicz condition can achieve 𝒪(√(p)/nϵ), which is the first 𝒪(1/n) high probability excess population risk bound w.r.t n for differentially private algorithms under non-smooth conditions. Moreover, we evaluate the performance of the new proposed algorithm m-NGP, the experimental results show that m-NGP improves the performance of the differentially private model over real datasets. It demonstrates that m-NGP improves the utility bound and the accuracy of the DP model on real datasets simultaneously.
READ FULL TEXT