research
∙
08/03/2023
Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
Going beyond stochastic gradient descent (SGD), what new phenomena emerg...
share
research
∙
06/12/2023
Transformers learn through gradual rank increase
We identify incremental learning dynamics in transformers, where the dif...
research
∙
05/22/2023
The NTK approximation is valid for longer than you think
We study when the neural tangent kernel (NTK) approximation is valid for...
research
∙
03/11/2023
Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Training stability is of great importance to Transformers. In this work,...
research
∙
06/10/2022
The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
The grokking phenomenon as reported by Power et al. ( arXiv:2201.02177 )...
research
∙
02/04/2022
Learning Representation from Neural Fisher Kernel with Low-rank Approximation
In this paper, we study the representation of neural networks from the v...
research
∙
07/02/2021
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
Deep linear networks trained with gradient descent yield low rank soluti...
research
∙
07/01/2021
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
We analyze the learning dynamics of infinitely wide neural networks with...
research
∙
05/08/2021
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at ini...
research
∙
06/13/2020
Collegial Ensembles
Modern neural network performance typically improves as model size incre...
research
∙
03/27/2020
On the Optimization Dynamics of Wide Hypernetworks
Recent results in the theoretical study of deep learning have shown that...
research
∙
01/28/2020
Residual Tangent Kernels
A recent body of work has focused on the theoretical study of neural net...
research
∙
01/14/2020
On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width
The Hessian of neural networks can be decomposed into a sum of two matri...
research
∙
11/21/2018
Regularizing by the Variance of the Activations' Sample-Variances
Normalization techniques play an important role in supporting efficient ...
research
∙
11/08/2016
The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization
Deep Residual Networks present a premium in performance in comparison to...
research
∙
11/29/2015