A mathematical theory of semantic development in deep neural networks

by   Andrew M. Saxe, et al.

An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: what are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep learning dynamics to give rise to these regularities.


page 1

page 8

page 10


Controlling Recurrent Neural Networks by Conceptors

The human brain is a dynamical system whose extremely complex sensor-dri...

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Despite the widespread practical success of deep learning methods, our t...

Introducing the structural bases of typicality effects in deep learning

In this paper, we hypothesize that the effects of the degree of typicali...

The Neural Race Reduction: Dynamics of Abstraction in Gated Networks

Our theoretical understanding of deep learning has not kept pace with it...

A Property Induction Framework for Neural Language Models

To what extent can experience from language contribute to our conceptual...

Criticality & Deep Learning I: Generally Weighted Nets

Motivated by the idea that criticality and universality of phase transit...

The Relativity of Induction

Lately there has been a lot of discussion about why deep learning algori...

Code Repositories


Understand the learning behavior of natural gradient descent vs SGD and other optimization methods

view repo