Adversarial Training Reduces Information and Improves Transferability

07/22/2020
by   Matteo Terzi, et al.
0

Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility. The latter property may seem counter-intuitive as it is widely accepted by the community that classification models should only capture the minimal information (features) required for the task. Motivated by this discrepancy, we investigate the dual relationship between Adversarial Training and Information Theory. We show that the Adversarial Training can improve linear transferability to new tasks, from which arises a new trade-off between transferability of representations and accuracy on the source task. We validate our results employing robust networks trained on CIFAR-10, CIFAR-100 and ImageNet on several datasets. Moreover, we show that Adversarial Training reduces Fisher information of representations about the input and of the weights about the task, and we provide a theoretical argument which explains the invertibility of deterministic networks without violating the principle of minimality. Finally, we leverage our theoretical insights to remarkably improve the quality of reconstructed images through inversion.

READ FULL TEXT

page 8

page 13

research
09/23/2019

Robust Local Features for Improving the Generalization of Adversarial Training

Adversarial training has been demonstrated as one of the most effective ...
research
06/18/2021

Adversarial Training Helps Transfer Learning via Better Representations

Transfer learning aims to leverage models pre-trained on source data to ...
research
02/09/2023

Better Diffusion Models Further Improve Adversarial Training

It has been recognized that the data generated by the denoising diffusio...
research
05/24/2022

One-Pixel Shortcut: on the Learning Preference of Deep Neural Networks

Unlearnable examples (ULEs) aim to protect data from unauthorized usage ...
research
09/26/2019

Towards Understanding the Transferability of Deep Representations

Deep neural networks trained on a wide range of datasets demonstrate imp...
research
10/10/2022

Revisiting adapters with adversarial training

While adversarial training is generally used as a defense mechanism, rec...
research
06/16/2020

Intriguing generalization and simplicity of adversarially trained neural networks

Adversarial training has been the topic of dozens of studies and a leadi...

Please sign up or login with your details

Forgot password? Click here to reset