The Lottery Ticket Hypothesis: Training Pruned Neural Networks

by   Jonathan Frankle, et al.

Recent work on neural network pruning indicates that, at training time, neural networks need to be significantly larger in size than is necessary to represent the eventual functions that they learn. This paper articulates a new hypothesis to explain this phenomenon. This conjecture, which we term the "lottery ticket hypothesis," proposes that successful training depends on lucky random initialization of a smaller subcomponent of the network. Larger networks have more of these "lottery tickets," meaning they are more likely to luck out with a subcomponent initialized in a configuration amenable to successful optimization. This paper conducts a series of experiments with XOR and MNIST that support the lottery ticket hypothesis. In particular, we identify these fortuitously-initialized subcomponents by pruning low-magnitude weights from trained networks. We then demonstrate that these subcomponents can be successfully retrained in isolation so long as the subnetworks are given the same initializations as they had at the beginning of the training process. Initialized as such, these small networks reliably converge successfully, often faster than the original network at the same level of accuracy. However, when these subcomponents are randomly reinitialized or rearranged, they perform worse than the original network. In other words, large networks that train successfully contain small subnetworks with initializations conducive to optimization. The lottery ticket hypothesis and its connection to pruning are a step toward developing architectures, initializations, and training strategies that make it possible to solve the same problems with much smaller networks.


page 1

page 2

page 3

page 4


The Lottery Ticket Hypothesis: Finding Small, Trainable Neural Networks

Neural network compression techniques are able to reduce the parameter c...

The Lottery Ticket Hypothesis at Scale

Recent work on the "lottery ticket hypothesis" proposes that randomly-in...

Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture

Sparse models require less memory for storage and enable a faster infere...

A Hypothesis for the Aesthetic Appreciation in Neural Networks

This paper proposes a hypothesis for the aesthetic appreciation that aes...

On the Existence of Universal Lottery Tickets

The lottery ticket hypothesis conjectures the existence of sparse subnet...

It's Hard for Neural Networks To Learn the Game of Life

Efforts to improve the learning abilities of neural networks have focuse...

Winning the Lottery with Continuous Sparsification

The Lottery Ticket Hypothesis from Frankle Carbin (2019) conjectures...

Please sign up or login with your details

Forgot password? Click here to reset