DeepAI AI Chat
Log In Sign Up

On Fragile Features and Batch Normalization in Adversarial Training

by   Nils Philipp Walter, et al.

Modern deep learning architecture utilize batch normalization (BN) to stabilize training and improve accuracy. It has been shown that the BN layers alone are surprisingly expressive. In the context of robustness against adversarial examples, however, BN is argued to increase vulnerability. That is, BN helps to learn fragile features. Nevertheless, BN is still used in adversarial training, which is the de-facto standard to learn robust features. In order to shed light on the role of BN in adversarial training, we investigate to what extent the expressiveness of BN can be used to robustify fragile features in comparison to random features. On CIFAR10, we find that adversarially fine-tuning just the BN layers can result in non-trivial adversarial robustness. Adversarially training only the BN layers from scratch, in contrast, is not able to convey meaningful adversarial robustness. Our results indicate that fragile features can be used to learn models with moderate adversarial robustness, while random features cannot


page 1

page 2

page 3

page 4


Towards the Desirable Decision Boundary by Moderate-Margin Adversarial Training

Adversarial training, as one of the most effective defense methods again...

Batch Normalization Increases Adversarial Vulnerability: Disentangling Usefulness and Robustness of Model Features

Batch normalization (BN) has been widely used in modern deep neural netw...

What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?

The adversarial vulnerability of neural nets, and subsequent techniques ...

Easy Batch Normalization

It was shown that adversarial examples improve object recognition. But w...

Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

Batch normalization (BatchNorm) has become an indispensable tool for tra...

Intriguing properties of adversarial training

Adversarial training is one of the main defenses against adversarial att...

The Expressive Power of Tuning Only the Norm Layers

Feature normalization transforms such as Batch and Layer-Normalization h...