Towards Understanding the Dynamics of the First-Order Adversaries

10/20/2020
by   Zhun Deng, et al.
0

An acknowledged weakness of neural networks is their vulnerability to adversarial perturbations to the inputs. To improve the robustness of these models, one of the most popular defense mechanisms is to alternatively maximize the loss over the constrained perturbations (or called adversaries) on the inputs using projected gradient ascent and minimize over weights. In this paper, we analyze the dynamics of the maximization step towards understanding the experimentally observed effectiveness of this defense mechanism. Specifically, we investigate the non-concave landscape of the adversaries for a two-layer neural network with a quadratic loss. Our main result proves that projected gradient ascent finds a local maximum of this non-concave problem in a polynomial number of iterations with high probability. To our knowledge, this is the first work that provides a convergence analysis of the first-order adversaries. Moreover, our analysis demonstrates that, in the initial phase of adversarial training, the scale of the inputs matters in the sense that a smaller input scale leads to faster convergence of adversarial training and a "more regular" landscape. Finally, we show that these theoretical findings are in excellent agreement with a series of experiments.

READ FULL TEXT

page 26

page 27

06/08/2018

Monge beats Bayes: Hardness Results for Adversarial Training

The last few years have seen extensive empirical study of the robustness...
08/01/2022

Attacking Adversarial Defences by Smoothing the Loss Landscape

This paper investigates a family of methods for defending against advers...
06/04/2022

Soft Adversarial Training Can Retain Natural Accuracy

Adversarial training for neural networks has been in the limelight in re...
05/27/2020

Enhancing Resilience of Deep Learning Networks by Means of Transferable Adversaries

Artificial neural networks in general and deep learning networks in part...
07/29/2020

Stylized Adversarial Defense

Deep Convolution Neural Networks (CNNs) can easily be fooled by subtle, ...
05/02/2019

You Only Propagate Once: Painless Adversarial Training Using Maximal Principle

Deep learning achieves state-of-the-art results in many areas. However r...
05/02/2019

You Only Propagate Once: Accelerating Adversarial Training Using Maximal Principle

Deep learning achieves state-of-the-art results in many areas. However r...