Scaling Up Influence Functions

12/06/2021
by   Andrea Schioppa, et al.
0

We address efficient calculation of influence functions for tracking predictions back to the training data. We propose and analyze a new approach to speeding up the inverse Hessian calculation based on Arnoldi iteration. With this improvement, we achieve, to the best of our knowledge, the first successful implementation of influence functions that scales to full-size (language and vision) Transformer models with several hundreds of millions of parameters. We evaluate our approach on image classification and sequence-to-sequence tasks with tens to a hundred of millions of training examples. Our code will be available at https://github.com/google-research/jax-influence.

READ FULL TEXT

page 12

page 13

page 15

page 16

page 17

research
12/31/2020

FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

Influence functions approximate the 'influences' of training data-points...
research
02/04/2021

HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks

The behaviors of deep neural networks (DNNs) are notoriously resistant t...
research
03/22/2021

Data Cleansing for Deep Neural Networks with Storage-efficient Approximation of Influence Functions

Identifying the influence of training data for data cleansing can improv...
research
11/03/2020

Shift If You Can: Counting and Visualising Correction Operations for Beat Tracking Evaluation

In this late-breaking abstract we propose a modified approach for beat t...
research
04/30/2022

Adapting and Evaluating Influence-Estimation Methods for Gradient-Boosted Decision Trees

Influence estimation analyzes how changes to the training data can lead ...
research
12/09/2022

Training Data Influence Analysis and Estimation: A Survey

Good models require good training data. For overparameterized deep model...
research
11/28/2022

Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality

Recent datasets expose the lack of the systematic generalization ability...

Please sign up or login with your details

Forgot password? Click here to reset