DeepAI AI Chat
Log In Sign Up

Towards Practical Lottery Ticket Hypothesis for Adversarial Training

03/06/2020
by   Bai Li, et al.
2

Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate training steps. While this discovery is insightful, finding proper sub-networks requires iterative training and pruning. The high cost incurred limits the applications of the lottery ticket hypothesis. We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process and thus can mitigate the cost issue. We conduct extensive experiments to show such sub-networks consistently exist across various model structures for a restrictive setting of hyperparameters (e.g., carefully selected learning rate, pruning ratio, and model capacity). As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training, a standard approach to improve robustness, by up to 49% on CIFAR-10 to achieve the state-of-the-art robustness.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/14/2022

Finding Dynamics Preserving Adversarial Winning Tickets

Modern deep neural networks (DNNs) are vulnerable to adversarial attacks...
03/29/2019

Second Rethinking of Network Pruning in the Adversarial Setting

It is well known that deep neural networks (DNNs) are vulnerable to adve...
02/24/2022

The rise of the lottery heroes: why zero-shot pruning is hard

Recent advances in deep learning optimization showed that just a subset ...
07/01/2021

Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?

There have been long-standing controversies and inconsistencies over the...
12/10/2019

Winning the Lottery with Continuous Sparsification

The Lottery Ticket Hypothesis from Frankle Carbin (2019) conjectures...
02/19/2021

Lottery Ticket Implies Accuracy Degradation, Is It a Desirable Phenomenon?

In deep model compression, the recent finding "Lottery Ticket Hypothesis...
03/08/2022

Dual Lottery Ticket Hypothesis

Fully exploiting the learning capacity of neural networks requires overp...