On the Accuracy of Influence Functions for Measuring Group Effects

05/30/2019
by   Pang Wei Koh, et al.
0

Influence functions estimate the effect of removing particular training points on a model without needing to retrain it. They are based on a first-order approximation that is accurate for small changes in the model, and so are commonly used for studying the effect of individual points in large datasets. However, we often want to study the effects of large groups of training points, e.g., to diagnose batch effect or apportion credit between different data sources. Removing such large groups can result in significant changes to the model. Are influence functions still accurate in this setting? In this paper, we find that across many different types of groups and in a range of real-world datasets, the influence of a group correlates surprisingly well with its actual effect, even if the absolute and relative error can be large. Our theoretical analysis shows that such correlation arises under certain settings but need not hold in general, indicating that real-world datasets have particular properties that keep the influence approximation well-behaved.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/01/2019

Second-Order Group Influence Functions for Black-Box Predictions

With the rapid adoption of machine learning systems in sensitive applica...
research
10/03/2022

Understanding Influence Functions and Datamodels via Harmonic Analysis

Influence functions estimate effect of individual data points on predict...
research
06/25/2020

Influence Functions in Deep Learning Are Fragile

Influence functions approximate the effect of training samples in test-t...
research
10/14/2022

Characterizing the Influence of Graph Elements

Influence function, a method from robust statistics, measures the change...
research
08/26/2021

Machine Unlearning of Features and Labels

Removing information from a machine learning model is a non-trivial task...
research
09/12/2022

If Influence Functions are the Answer, Then What is the Question?

Influence functions efficiently estimate the effect of removing a single...
research
12/20/2022

Improving the Robustness of Summarization Models by Detecting and Removing Input Noise

The evaluation of abstractive summarization models typically uses test d...

Please sign up or login with your details

Forgot password? Click here to reset