Etai Littwin | DeepAI

DeepAI AI Chat

Featured Co-authors

Lior Wolf
168 publications
Yizhe Zhang
80 publications
Jiatao Gu
61 publications
Samy Bengio
55 publications
Emmanuel Abbe
37 publications
Shuangfei Zhai
28 publications
Tatiana Likhomanenko
26 publications
Greg Yang
25 publications
Josh Susskind
20 publications
Tomer Galanti
20 publications
Jason Ramapuram
18 publications

∙ 08/03/2023

Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit

Going beyond stochastic gradient descent (SGD), what new phenomena emerg...

0 Greg Yang, et al. ∙

∙ 06/12/2023

Transformers learn through gradual rank increase

We identify incremental learning dynamics in transformers, where the dif...

0 Enric Boix-Adserà, et al. ∙

∙ 05/22/2023

The NTK approximation is valid for longer than you think

We study when the neural tangent kernel (NTK) approximation is valid for...

0 Enric Boix-Adserà, et al. ∙

∙ 03/11/2023

Stabilizing Transformer Training by Preventing Attention Entropy Collapse

Training stability is of great importance to Transformers. In this work,...

5 Shuangfei Zhai, et al. ∙

∙ 06/10/2022

The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon

The grokking phenomenon as reported by Power et al. ( arXiv:2201.02177 )...

13 Vimal Thilak, et al. ∙

∙ 02/04/2022

Learning Representation from Neural Fisher Kernel with Low-rank Approximation

In this paper, we study the representation of neural networks from the v...

17 Ruixiang Zhang, et al. ∙

∙ 07/02/2021

Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks

Deep linear networks trained with gradient descent yield low rank soluti...

0 Shih-Yu Sun, et al. ∙

∙ 07/01/2021

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

We analyze the learning dynamics of infinitely wide neural networks with...

4 Etai Littwin, et al. ∙

∙ 05/08/2021

Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics

Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at ini...

2 Greg Yang, et al. ∙

∙ 06/13/2020

Collegial Ensembles

Modern neural network performance typically improves as model size incre...

0 Etai Littwin, et al. ∙

∙ 03/27/2020

On the Optimization Dynamics of Wide Hypernetworks

Recent results in the theoretical study of deep learning have shown that...

2 Etai Littwin, et al. ∙

∙ 01/28/2020

Residual Tangent Kernels

A recent body of work has focused on the theoretical study of neural net...

0 Etai Littwin, et al. ∙

∙ 01/14/2020

On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width

The Hessian of neural networks can be decomposed into a sum of two matri...

0 Etai Littwin, et al. ∙

∙ 11/21/2018

Regularizing by the Variance of the Activations' Sample-Variances

Normalization techniques play an important role in supporting efficient ...

0 Etai Littwin, et al. ∙

∙ 11/08/2016

The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization

Deep Residual Networks present a premium in performance in comparison to...

0 Etai Littwin, et al. ∙

∙ 11/29/2015

The Multiverse Loss for Robust Transfer Learning

Deep learning techniques are renowned for supporting effective transfer ...

0 Etai Littwin, et al. ∙

Are you a researcher?
Expose your work
to one of the largest
A.I. communities in the world