DeepAI AI Chat
Log In Sign Up

Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient

by   Ankit Pensia, et al.
University of Wisconsin-Madison

The strong lottery ticket hypothesis (LTH) postulates that one can approximate any target neural network by only pruning the weights of a sufficiently over-parameterized random network. A recent work by Malach et al. <cit.> establishes the first theoretical analysis for the strong LTH: one can provably approximate a neural network of width d and depth l, by pruning a random one that is a factor O(d^4l^2) wider and twice as deep. This polynomial over-parameterization requirement is at odds with recent experimental research that achieves good approximation with networks that are a small factor wider than the target. In this work, we close the gap and offer an exponential improvement to the over-parameterization requirement for the existence of lottery tickets. We show that any target network of width d and depth l can be approximated by pruning a random network that is a factor O(log(dl)) wider and twice as deep. Our analysis heavily relies on connecting pruning random ReLU networks to random instances of the SubsetSum problem. We then show that this logarithmic over-parameterization is essentially optimal for constant depth networks. Finally, we verify several of our theoretical insights with experiments.


page 1

page 2

page 3

page 4


Finding Everything within Random Binary Networks

A recent work by Ramanujan et al. (2020) provides significant empirical ...

How Erdös and Rényi Win the Lottery

Random masks define surprisingly effective sparse neural network models,...

Strong Lottery Ticket Hypothesis with ε–perturbation

The strong Lottery Ticket Hypothesis (LTH) claims the existence of a sub...

Logarithmic Pruning is All You Need

The Lottery Ticket Hypothesis is a conjecture that every large neural ne...

A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis

The Strong Lottery Ticket Hypothesis (SLTH) stipulates the existence of ...

On the Neural Tangent Kernel Analysis of Randomly Pruned Wide Neural Networks

We study the behavior of ultra-wide neural networks when their weights a...

Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem

Given a target function f, how large must a neural network be in order t...