research
∙
09/11/2023
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the re...
share
research
∙
01/05/2023
Training trajectories, mini-batch losses and the curious role of the learning rate
Stochastic gradient descent plays a fundamental role in nearly all appli...
research
∙
11/28/2022
Decentralized Learning with Multi-Headed Distillation
Decentralized learning with private data is a central problem in machine...
research
∙
04/10/2021
