High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Data

07/23/2021
by   Lijie Hu, et al.
0

As one of the most fundamental problems in machine learning, statistics and differential privacy, Differentially Private Stochastic Convex Optimization (DP-SCO) has been extensively studied in recent years. However, most of the previous work can only handle either regular data distribution or irregular data in the low dimensional space case. To better understand the challenges arising from irregular data distribution, in this paper we provide the first study on the problem of DP-SCO with heavy-tailed data in the high dimensional space. In the first part we focus on the problem over some polytope constraint (such as the ℓ_1-norm ball). We show that if the loss function is smooth and its gradient has bounded second order moment, it is possible to get a (high probability) error bound (excess population risk) of Õ(log d/(nϵ)^1/3) in the ϵ-DP model, where n is the sample size and d is the dimensionality of the underlying space. Next, for LASSO, if the data distribution that has bounded fourth-order moments, we improve the bound to Õ(log d/(nϵ)^2/5) in the (ϵ, δ)-DP model. In the second part of the paper, we study sparse learning with heavy-tailed data. We first revisit the sparse linear model and propose a truncated DP-IHT method whose output could achieve an error of Õ(s^*2log d/nϵ), where s^* is the sparsity of the underlying parameter. Then we study a more general problem over the sparsity (i.e., ℓ_0-norm) constraint, and show that it is possible to achieve an error of Õ(s^*3/2log d/nϵ), which is also near optimal up to a factor of Õ(√(s^*)), if the loss function is smooth and strongly convex.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/10/2022

Differentially Private ℓ_1-norm Linear Regression with Heavy-tailed Data

We study the problem of Differentially Private Stochastic Convex Optimiz...
research
10/21/2020

On Differentially Private Stochastic Convex Optimization with Heavy-tailed Data

In this paper, we consider the problem of designing Differentially Priva...
research
03/31/2023

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited

In this paper, we revisit the problem of Differentially Private Stochast...
research
06/02/2021

Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data

We study stochastic convex optimization with heavy-tailed data under the...
research
06/27/2022

Efficient Private SCO for Heavy-Tailed Data via Clipping

We consider stochastic convex optimization for heavy-tailed data with th...
research
11/20/2014

Private Empirical Risk Minimization Beyond the Worst Case: The Effect of the Constraint Set Geometry

Empirical Risk Minimization (ERM) is a standard technique in machine lea...
research
09/16/2022

Truthful Generalized Linear Models

In this paper we study estimating Generalized Linear Models (GLMs) in th...

Please sign up or login with your details

Forgot password? Click here to reset