Usable Information and Evolution of Optimal Representations During Training

10/06/2020
by   Michael Kleinman, et al.
0

We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training, and how they adapt to different tasks. We use this to characterize the transient dynamics of deep neural networks on perceptual decision-making tasks inspired by neuroscience literature. In particular, we show that both the random initialization and the implicit regularization from Stochastic Gradient Descent play an important role in learning minimal sufficient representations for the task. If the network is not randomly initialized, we show that the training may not recover an optimal representation, increasing the chance of overfitting.

READ FULL TEXT

page 13

page 14

research
03/05/2019

Implicit Regularization in Over-parameterized Neural Networks

Over-parameterized neural networks generalize well in practice without a...
research
01/07/2019

Generalization in Deep Networks: The Role of Distance from Initialization

Why does training deep neural networks using stochastic gradient descent...
research
10/14/2020

Deep Neural Network Training with Frank-Wolfe

This paper studies the empirical efficacy and benefits of using projecti...
research
03/22/2022

Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals

Humans can learn several tasks in succession with minimal mutual interfe...
research
02/19/2021

On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent

Recent work has highlighted the role of initialization scale in determin...
research
06/30/2020

Maximum Entropy Models for Fast Adaptation

Deep Neural Networks have shown great promise on a variety of downstream...
research
11/21/2022

Representational dissimilarity metric spaces for stochastic neural networks

Quantifying similarity between neural representations – e.g. hidden laye...

Please sign up or login with your details

Forgot password? Click here to reset