Towards Practical Lottery Ticket Hypothesis for Adversarial Training

03/06/2020
by   Bai Li, et al.
2

Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate training steps. While this discovery is insightful, finding proper sub-networks requires iterative training and pruning. The high cost incurred limits the applications of the lottery ticket hypothesis. We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process and thus can mitigate the cost issue. We conduct extensive experiments to show such sub-networks consistently exist across various model structures for a restrictive setting of hyperparameters (e.g., carefully selected learning rate, pruning ratio, and model capacity). As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training, a standard approach to improve robustness, by up to 49% on CIFAR-10 to achieve the state-of-the-art robustness.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/14/2022

Finding Dynamics Preserving Adversarial Winning Tickets

Modern deep neural networks (DNNs) are vulnerable to adversarial attacks...
research
12/05/2019

The Search for Sparse, Robust Neural Networks

Recent work on deep neural network pruning has shown there exist sparse ...
research
02/24/2022

The rise of the lottery heroes: why zero-shot pruning is hard

Recent advances in deep learning optimization showed that just a subset ...
research
09/08/2021

Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?

The proposition of lottery ticket hypothesis revealed the relationship b...
research
07/01/2021

Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?

There have been long-standing controversies and inconsistencies over the...
research
02/19/2021

Lottery Ticket Implies Accuracy Degradation, Is It a Desirable Phenomenon?

In deep model compression, the recent finding "Lottery Ticket Hypothesis...
research
12/10/2019

Winning the Lottery with Continuous Sparsification

The Lottery Ticket Hypothesis from Frankle Carbin (2019) conjectures...

Please sign up or login with your details

Forgot password? Click here to reset