The Lottery Ticket Hypothesis at Scale

by   Jonathan Frankle, et al.

Recent work on the "lottery ticket hypothesis" proposes that randomly-initialized, dense neural networks contain much smaller, fortuitously initialized subnetworks ("winning tickets") capable of training to similar accuracy as the original network at a similar speed. While strong evidence exists for the hypothesis across many settings, it has not yet been evaluated on large, state-of-the-art networks and there is even evidence against the hypothesis on deeper networks. We modify the lottery ticket pruning procedure to make it possible to identify winning tickets on deeper networks. Rather than set the weights of a winning ticket to their original initializations, we set them to the weights obtained after a small number of training iterations ("late resetting"). Using late resetting, we identify the first winning tickets for Resnet-50 on Imagenet To understand the efficacy of late resetting, we study the "stability" of neural network training to pruning, which we define as the consistency of the optimization trajectories followed by a winning ticket when it is trained in isolation and as part of the larger network. We find that later resetting produces stabler winning tickets and that improved stability correlates with higher winning ticket accuracy. This analysis offers new insights into the lottery ticket hypothesis and the dynamics of neural network learning.


page 1

page 2

page 3

page 4


The Lottery Ticket Hypothesis: Training Pruned Neural Networks

Recent work on neural network pruning indicates that, at training time, ...

Proving the Lottery Ticket Hypothesis: Pruning is All You Need

The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a ...

How many winning tickets are there in one DNN?

The recent lottery ticket hypothesis proposes that there is one sub-netw...

Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?

The proposition of lottery ticket hypothesis revealed the relationship b...

Linear Mode Connectivity and the Lottery Ticket Hypothesis

We introduce "instability analysis," a framework for assessing whether t...

Finding Everything within Random Binary Networks

A recent work by Ramanujan et al. (2020) provides significant empirical ...

Strong Lottery Ticket Hypothesis with ε–perturbation

The strong Lottery Ticket Hypothesis (LTH) claims the existence of a sub...

Please sign up or login with your details

Forgot password? Click here to reset