Measuring Information Transfer in Neural Networks

09/16/2020
by   Xiao Zhang, et al.
10

Estimation of the information content in a neural network model can be prohibitive, because of difficulty in finding an optimal codelength of the model. We propose to use a surrogate measure to bypass directly estimating model information. The proposed Information Transfer (L_IT) is a measure of model information based on prequential coding. L_IT is theoretically connected to model information, and is consistently correlated with model information in experiments. We show that L_IT can be used as a measure of generalizable knowledge in a model or a dataset. Therefore, L_IT can serve as an analytical tool in deep learning. We apply L_IT to compare and dissect information in datasets, evaluate representation models in transfer learning, and analyze catastrophic forgetting and continual learning algorithms. L_IT provides an informational perspective which helps us discover new insights into neural network learning.

READ FULL TEXT
research
03/14/2023

Is forgetting less a good inductive bias for forward transfer?

One of the main motivations of studying continual learning is that the p...
research
08/12/2022

Communication network model of the immune system identifies the impact of interactions with SARS-CoV-2 proteins

Interactions between SARS-CoV-2 and human proteins (SARS-CoV-2 PPIs) cau...
research
10/11/2022

Continual Learning by Modeling Intra-Class Variation

It has been observed that neural networks perform poorly when the data o...
research
06/24/2019

Lifelong Learning Starting From Zero

We present a deep neural-network model for lifelong learning inspired by...
research
06/01/2022

Transfer without Forgetting

This work investigates the entanglement between Continual Learning (CL) ...
research
02/09/2022

A Neural Network Model of Continual Learning with Cognitive Control

Neural networks struggle in continual learning settings from catastrophi...
research
12/10/2019

Reducing Catastrophic Forgetting in Modular Neural Networks by Dynamic Information Balancing

Lifelong learning is a very important step toward realizing robust auton...

Please sign up or login with your details

Forgot password? Click here to reset