AI Chat AI Image Generator AI Video Text to Speech

Visualizing the Loss Landscape of Winning Lottery Tickets

12/16/2021

∙

by Robert Bain, et al.

∙

∙

The underlying loss landscapes of deep neural networks have a great impact on their training, but they have mainly been studied theoretically due to computational constraints. This work vastly reduces the time required to compute such loss landscapes, and uses them to study winning lottery tickets found via iterative magnitude pruning. We also share results that contradict previously claimed correlations between certain loss landscape projection methods and model trainability and generalization error.

page 1

page 2

page 3

page 4

page 6

page 7

research

∙ 07/22/2023

The instabilities of large learning rate training: a loss landscape view

Modern neural networks are undeniably successful. Numerous works study h...

0 Lawrence Wang, et al. ∙

research

∙ 02/05/2021

Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression

Adversarial training is actively studied for learning robust models agai...

0 Masanori Yamada, et al. ∙

research

∙ 06/30/2017

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

It is widely observed that deep learning models with learned parameters ...

0 Lei Wu, et al. ∙

research

∙ 05/26/2022

Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks

Unraveling the general structure underlying the loss landscapes of deep ...

0 Zhiwei Bai, et al. ∙

research

∙ 05/20/2023

Loss Spike in Training Neural Networks

In this work, we study the mechanism underlying loss spikes observed dur...

0 Zhongwang Zhang, et al. ∙

research

∙ 03/19/2018

Comparing Dynamics: Deep Neural Networks versus Glassy Systems

We analyze numerically the training dynamics of deep neural networks (DN...

0 M. Baity-Jesi, et al. ∙

research

∙ 06/23/2023

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

This paper focuses on predicting the occurrence of grokking in neural ne...

0 Pascal Jr. Tikeng Notsawo, et al. ∙