Critical Learning Periods in Deep Neural Networks
share
Critical periods are phases in the early development of humans and animals during which experience can affect the structure of neuronal networks irreversibly. In this work, we study the effects of visual stimulus deficits on the training of artificial neural networks (ANNs). Introducing well-characterized visual deficits, such as cataract-like blurring, in the early training phase of a standard deep neural network causes irreversible performance loss that closely mimics that reported in humans and animal models. Deficits that do not affect low-level image statistics, such as vertical flipping of the images, have no lasting effect on the ANN's performance and can be rapidly overcome with additional training, as observed in humans. In addition, deeper networks show a more prominent critical period. To better understand this phenomenon, we use techniques from information theory to study the strength of the network connections during training. Our analysis suggests that the first few epochs are critical for the allocation of resources across different layers, determined by the initial input data distribution. Once such information organization is established, the network resources do not re-distribute through additional training. These findings suggest that the initial rapid learning phase of training of ANNs, under-scrutinized compared to its asymptotic behavior, plays a key role in defining the final performance of networks.
READ FULL TEXT