research
∙
06/07/2023
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
In this paper, we first present an explanation regarding the common occu...
share
research
∙
09/29/2022
Restricted Strong Convexity of Deep Learning Models with Smooth Activations
We consider the problem of optimization of deep learning models with smo...
research
∙
06/30/2022
A note on Linear Bottleneck networks and their Transition to Multilinearity
Randomly initialized wide neural networks transition to linear functions...
research
∙
05/24/2022
Quadratic models for understanding neural network dynamics
In this work, we propose using a quadratic model as a tool for understan...
research
∙
05/24/2022
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
In this paper we show that feedforward neural networks corresponding to ...
research
∙
03/10/2022
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Wide neural networks with linear output layer have been shown to be near...
research
∙
10/02/2020
On the linearity of large non-linear models: when and why the tangent kernel is constant
The goal of this work is to shed light on the remarkable phenomenon of t...
research
∙
02/29/2020
