research
∙
06/05/2023
Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking
The backtracking line-search is an effective technique to automatically ...
share
research
∙
04/27/2023
Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
The success of the Adam optimizer on a wide array of architectures has m...
research
∙
11/12/2021
Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent – an Open Problem
We consider the problem of upper bounding the expected log-likelihood su...
research
∙
11/02/2020
Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent
Expectation maximization (EM) is the default algorithm for fitting proba...
research
∙
06/11/2020
Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search)
As adaptive gradient methods are typically used for training over-parame...
research
∙
12/23/2019
BackPACK: Packing more into backprop
Automatic differentiation frameworks are optimized for exactly one thing...
research
∙
05/29/2019
Limitations of the Empirical Fisher Approximation
Natural gradient descent, which preconditions a gradient descent update ...
research
∙
11/11/2018
