Critical Learning Periods in Deep Neural Networks

11/24/2017
by   Alessandro Achille, et al.
0

Critical periods are phases in the early development of humans and animals during which experience can affect the structure of neuronal networks irreversibly. In this work, we study the effects of visual stimulus deficits on the training of artificial neural networks (ANNs). Introducing well-characterized visual deficits, such as cataract-like blurring, in the early training phase of a standard deep neural network causes irreversible performance loss that closely mimics that reported in humans and animal models. Deficits that do not affect low-level image statistics, such as vertical flipping of the images, have no lasting effect on the ANN's performance and can be rapidly overcome with additional training, as observed in humans. In addition, deeper networks show a more prominent critical period. To better understand this phenomenon, we use techniques from information theory to study the strength of the network connections during training. Our analysis suggests that the first few epochs are critical for the allocation of resources across different layers, determined by the initial input data distribution. Once such information organization is established, the network resources do not re-distribute through additional training. These findings suggest that the initial rapid learning phase of training of ANNs, under-scrutinized compared to its asymptotic behavior, plays a key role in defining the final performance of networks.

READ FULL TEXT
research
10/06/2022

Critical Learning Periods for Multisensory Integration in Deep Networks

We show that the ability of a neural network to integrate information fr...
research
08/23/2023

Critical Learning Periods Emerge Even in Deep Linear Networks

Critical learning periods are periods early in development where tempora...
research
02/21/2020

The Break-Even Point on Optimization Trajectories of Deep Neural Networks

The early phase of training of deep neural networks is critical for thei...
research
07/17/2022

Improving Deep Neural Network Random Initialization Through Neuronal Rewiring

The deep learning literature is continuously updated with new architectu...
research
09/12/2021

Critical Learning Periods in Federated Learning

Federated learning (FL) is a popular technique to train machine learning...
research
06/26/2017

Do Deep Neural Networks Suffer from Crowding?

Crowding is a visual effect suffered by humans, in which an object that ...
research
12/02/2019

Simulation of neural function in an artificial Hebbian network

Artificial neural networks have diverged far from their early inspiratio...

Please sign up or login with your details

Forgot password? Click here to reset