Using Mode Connectivity for Loss Landscape Analysis

06/18/2018
by   Akhilesh Gotmare, et al.
0

Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trained or initialized differently. We find that the procedure is resilient to such changes. Given this finding, we propose using the framework for analyzing loss surfaces and training trajectories more generally, and in this direction, study SGD with cosine annealing and restarts (SGDR). We report that while SGDR moves over barriers in its trajectory, propositions claiming that it converges to and escapes from multiple local minima are not substantiated by our empirical results.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/25/2022

Exploring Mode Connectivity for Pre-trained Language Models

Recent years have witnessed the prevalent application of pre-trained lan...
research
09/05/2020

Optimizing Mode Connectivity via Neuron Alignment

The loss landscapes of deep neural networks are not well understood due ...
research
02/25/2021

Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling

With a better understanding of the loss surfaces for multilayer networks...
research
10/13/2022

Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks

Based on the concepts of Wasserstein barycenter (WB) and Gromov-Wasserst...
research
01/08/2019

Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions

Quantification of the stationary points and the associated basins of att...
research
06/26/2023

Black holes and the loss landscape in machine learning

Understanding the loss landscape is an important problem in machine lear...
research
06/14/2021

Revisiting Model Stitching to Compare Neural Representations

We revisit and extend model stitching (Lenc Vedaldi 2015) as a metho...

Please sign up or login with your details

Forgot password? Click here to reset