Information, Privacy and Stability in Adaptive Data Analysis

06/02/2017
by   Adam Smith, et al.
0

Traditional statistical theory assumes that the analysis to be performed on a given data set is selected independently of the data themselves. This assumption breaks downs when data are re-used across analyses and the analysis to be performed at a given stage depends on the results of earlier stages. Such dependency can arise when the same data are used by several scientific studies, or when a single analysis consists of multiple stages. How can we draw statistically valid conclusions when data are re-used? This is the focus of a recent and active line of work. At a high level, these results show that limiting the information revealed by earlier stages of analysis controls the bias introduced in later stages by adaptivity. Here we review some known results in this area and highlight the role of information-theoretic concepts, notably several one-shot notions of mutual information.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/16/2015

How much does your data exploration overfit? Controlling bias via information usage

Modern data is messy and high-dimensional, and it is often not clear a p...
research
09/08/2021

A Bayesian Framework for Information-Theoretic Probing

Pimentel et al. (2020) recently analysed probing from an information-the...
research
01/24/2020

Reasoning About Generalization via Conditional Mutual Information

We provide an information-theoretic framework for studying the generaliz...
research
06/16/2016

Estimating mutual information in high dimensions via classification error

Multivariate pattern analyses approaches in neuroimaging are fundamental...
research
09/15/2023

Adaptive Neyman Allocation

In experimental design, Neyman allocation refers to the practice of allo...
research
10/17/2020

MithraDetective: A System for Cherry-picked Trendlines Detection

Given a data set, misleading conclusions can be drawn from it by cherry-...

Please sign up or login with your details

Forgot password? Click here to reset