
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at ini...
read it

Rapid Feature Evolution Accelerates Learning in Neural Networks
Neural network (NN) training and generalization in the infinitewidth li...
read it

Dynamically Stable InfiniteWidth Limits of Neural Classifiers
Recent research has been focused on two different approaches to studying...
read it

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
We analyze the learning dynamics of infinitely wide neural networks with...
read it

Towards a General Theory of InfiniteWidth Limits of Neural Classifiers
Obtaining theoretical guarantees for neural networks training appears to...
read it

A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs
Deep neural networks (DNNs) in the infinite width/channel limit have rec...
read it

Tensor Programs II: Neural Tangent Kernel for Any Architecture
We prove that a randomly initialized neural network of *any architecture...
read it
Feature Learning in InfiniteWidth Neural Networks
As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the NTK parametrization). However, we show that the standard and NTK parametrizations of a neural network do not admit infinitewidth limits that can learn features, which is crucial for pretraining and transfer learning such as with BERT. We propose simple modifications to the standard parametrization to allow for feature learning in the limit. Using the *Tensor Programs* technique, we derive explicit formulas for such limits. On Word2Vec and fewshot learning on Omniglot via MAML, two canonical tasks that rely crucially on feature learning, we compute these limits exactly. We find that they outperform both NTK baselines and finitewidth networks, with the latter approaching the infinitewidth feature learning performance as width increases. More generally, we classify a natural space of neural network parametrizations that generalizes standard, NTK, and Mean Field parametrizations. We show 1) any parametrization in this space either admits feature learning or has an infinitewidth training dynamics given by kernel gradient descent, but not both; 2) any such infinitewidth limit can be computed using the Tensor Programs technique.
READ FULL TEXT
Comments
There are no comments yet.