Estimating informativeness of samples with Smooth Unique Information

01/17/2021
by   Hrayr Harutyunyan, et al.
6

We define a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the final weights and how much it informs the function computed by the weights. Though related, we show that these quantities have a qualitatively different behavior. We give efficient approximations of these quantities using a linearized network and demonstrate empirically that the approximation is accurate for real-world architectures, such as pre-trained ResNets. We apply these measures to several problems, such as dataset summarization, analysis of under-sampled classes, comparison of informativeness of different data sources, and detection of adversarial and corrupted examples. Our work generalizes existing frameworks but enjoys better computational properties for heavily over-parametrized models, which makes it possible to apply it to real-world networks.

READ FULL TEXT

page 13

page 15

11/09/2017

Material Classification in the Wild: Do Synthesized Training Data Generalise Better than Real-World Training Data?

We question the dominant role of real-world training images in the field...
02/08/2019

Fourier Neural Networks: A Comparative Study

We review neural network architectures which were motivated by Fourier s...
10/10/2019

Learning from Multiple Corrupted Sources, with Application to Learning from Label Proportions

We study binary classification in the setting where the learner is prese...
07/13/2018

Unique Informations and Deficiencies

Given two channels that convey information about the same random variabl...
06/05/2018

The Value of Information in Retrospect

In the course of any statistical analysis, it is necessary to consider i...
05/29/2019

Where is the Information in a Deep Neural Network?

Whatever information a Deep Neural Network has gleaned from past data is...
09/02/2020

Adversarially Robust Neural Architectures

Deep Neural Network (DNN) are vulnerable to adversarial attack. Existing...

Code Repositories

aws-cv-unique-information

We define and estimate smooth unique information of samples with respect to classifier weights and predictions. We compute these quantities for linearized neural networks.


view repo