Fast Adversarial Training with Adaptive Step Size

06/06/2022
by   Zhichao Huang, et al.
5

While adversarial training and its variants have shown to be the most effective algorithms to defend against adversarial attacks, their extremely slow training process makes it hard to scale to large datasets like ImageNet. The key idea of recent works to accelerate adversarial training is to substitute multi-step attacks (e.g., PGD) with single-step attacks (e.g., FGSM). However, these single-step methods suffer from catastrophic overfitting, where the accuracy against PGD attack suddenly drops to nearly 0 training, destroying the robustness of the networks. In this work, we study the phenomenon from the perspective of training instances. We show that catastrophic overfitting is instance-dependent and fitting instances with larger gradient norm is more likely to cause catastrophic overfitting. Based on our findings, we propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS). ATAS learns an instancewise adaptive step size that is inversely proportional to its gradient norm. The theoretical analysis shows that ATAS converges faster than the commonly adopted non-adaptive counterparts. Empirically, ATAS consistently mitigates catastrophic overfitting and achieves higher robust accuracy on CIFAR10, CIFAR100 and ImageNet when evaluated on various adversarial budgets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/05/2020

Understanding Catastrophic Overfitting in Single-step Adversarial Training

Adversarial examples are perturbed inputs that are designed to deceive m...
research
06/30/2021

Single-Step Adversarial Training for Semantic Segmentation

Even though deep neural networks succeed on many different tasks includi...
research
02/23/2023

Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective

Although fast adversarial training provides an efficient approach for bu...
research
03/29/2021

ZeroGrad : Mitigating and Explaining Catastrophic Overfitting in FGSM Adversarial Training

Making deep neural networks robust to small adversarial noises has recen...
research
07/18/2022

Prior-Guided Adversarial Initialization for Fast Adversarial Training

Fast adversarial training (FAT) effectively improves the efficiency of s...
research
11/18/2019

WITCHcraft: Efficient PGD attacks with random step size

State-of-the-art adversarial attacks on neural networks use expensive it...

Please sign up or login with your details

Forgot password? Click here to reset