On Differentially Private Stochastic Convex Optimization with Heavy-tailed Data

10/21/2020
by   Di Wang, et al.
0

In this paper, we consider the problem of designing Differentially Private (DP) algorithms for Stochastic Convex Optimization (SCO) on heavy-tailed data. The irregularity of such data violates some key assumptions used in almost all existing DP-SCO and DP-ERM methods, resulting in failure to provide the DP guarantees. To better understand this type of challenges, we provide in this paper a comprehensive study of DP-SCO under various settings. First, we consider the case where the loss function is strongly convex and smooth. For this case, we propose a method based on the sample-and-aggregate framework, which has an excess population risk of Õ(d^3/nϵ^4) (after omitting other factors), where n is the sample size and d is the dimensionality of the data. Then, we show that with some additional assumptions on the loss functions, it is possible to reduce the expected excess population risk to Õ( d^2/ nϵ^2 ). To lift these additional conditions, we also provide a gradient smoothing and trimming based scheme to achieve excess population risks of Õ( d^2/nϵ^2) and Õ(d^2/3/(nϵ^2)^1/3) for strongly convex and general convex loss functions, respectively, with high probability. Experiments suggest that our algorithms can effectively deal with the challenges caused by data irregularity.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/31/2021

Faster Rates of Differentially Private Stochastic Convex Optimization

In this paper, we revisit the problem of Differentially Private Stochast...
research
03/31/2023

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited

In this paper, we revisit the problem of Differentially Private Stochast...
research
06/27/2022

Efficient Private SCO for Heavy-Tailed Data via Clipping

We consider stochastic convex optimization for heavy-tailed data with th...
research
06/02/2021

Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data

We study stochastic convex optimization with heavy-tailed data under the...
research
07/23/2021

High Dimensional Differentially Private Stochastic Optimization with Heavy-tailed Data

As one of the most fundamental problems in machine learning, statistics ...
research
01/10/2022

Differentially Private ℓ_1-norm Linear Regression with Heavy-tailed Data

We study the problem of Differentially Private Stochastic Convex Optimiz...

Please sign up or login with your details

Forgot password? Click here to reset