DeepAI AI Chat
Log In Sign Up

Git Re-Basin: Merging Models modulo Permutation Symmetries

by   Samuel K. Ainsworth, et al.
University of Washington

The success of deep learning is thanks to our ability to solve certain massive non-convex optimization problems with relative ease. Despite non-convex optimization being NP-hard, simple algorithms – often variants of stochastic gradient descent – exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes contain (nearly) a single basin, after accounting for all possible permutation symmetries of hidden units. We introduce three algorithms to permute the units of one model to bring them into alignment with units of a reference model. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10 and CIFAR-100. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity across a variety of models and datasets. Finally, we discuss shortcomings of a single basin theory, including a counterexample to the linear mode connectivity hypothesis.


Non-convex Optimization for Machine Learning

A vast majority of machine learning algorithms train their models and pe...

Why Do Local Methods Solve Nonconvex Problems?

Non-convex optimization is ubiquitous in modern machine learning. Resear...

Train faster, generalize better: Stability of stochastic gradient descent

We show that parametric models trained by a stochastic gradient method (...

Qualitatively characterizing neural network optimization problems

Training neural networks involves solving large-scale non-convex optimiz...

Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process

We consider deep networks, trained via stochastic gradient descent to mi...

Mode Connectivity in Auction Design

Optimal auction design is a fundamental problem in algorithmic game theo...

Code Repositories


Code release for "Git Re-Basin: Merging Models modulo Permutation Symmetries"

view repo