A New Analysis of Differential Privacy's Generalization Guarantees

by   Christopher Jung, et al.

We give a new proof of the "transfer theorem" underlying adaptive data analysis: that any mechanism for answering adaptively chosen statistical queries that is differentially private and sample-accurate is also accurate out-of-sample. Our new proof is elementary and gives structural insights that we expect will be useful elsewhere. We show: 1) that differential privacy ensures that the expectation of any query on the posterior distribution on datasets induced by the transcript of the interaction is close to its true value on the data distribution, and 2) sample accuracy on its own ensures that any query answer produced by the mechanism is close to its posterior expectation with high probability. This second claim follows from a thought experiment in which we imagine that the dataset is resampled from the posterior distribution after the mechanism has committed to its answers. The transfer theorem then follows by summing these two bounds, and in particular, avoids the "monitor argument" used to derive high probability bounds in prior work. An upshot of our new proof technique is that the concrete bounds we obtain are substantially better than the best previously known bounds, even though the improvements are in the constants, rather than the asymptotics (which are known to be tight). As we show, our new bounds outperform the naive "sample-splitting" baseline at dramatically smaller dataset sizes compared to the previous state of the art, bringing techniques from this literature closer to practicality.


page 1

page 2

page 3

page 4


Prior-Aware Distribution Estimation for Differential Privacy

Joint distribution estimation of a dataset under differential privacy is...

Calibrating Noise to Variance in Adaptive Data Analysis

Datasets are often used multiple times and each successive analysis may ...

Generalization for Adaptively-chosen Estimators via Stable Median

Datasets are often reused to perform multiple statistical analyses in an...

On Avoiding the Union Bound When Answering Multiple Differentially Private Queries

In this work, we study the problem of answering k queries with (ϵ, δ)-di...

Rényi Differential Privacy Mechanisms for Posterior Sampling

Using a recently proposed privacy definition of Rényi Differential Priva...

Bayesian Adaptive Data Analysis Guarantees from Subgaussianity

The new field of adaptive data analysis seeks to provide algorithms and ...

A Note on High-Probability versus In-Expectation Guarantees of Generalization Bounds in Machine Learning

Statistical machine learning theory often tries to give generalization g...

Please sign up or login with your details

Forgot password? Click here to reset