Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness

04/30/2020
by   Pu Zhao, et al.
7

Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural networks. In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. When network models are tampered with backdoor or error-injection attacks, our results demonstrate that the path connection learned using limited amount of bonafide data can effectively mitigate adversarial effects while maintaining the original accuracy on clean data. Therefore, mode connectivity provides users with the power to repair backdoored or error-injected models. We also use mode connectivity to investigate the loss landscapes of regular and robust models against evasion attacks. Experiments show that there exists a barrier in adversarial robustness loss on the path connecting regular and adversarially-trained models. A high correlation is observed between the adversarial robustness loss and the largest eigenvalue of the input Hessian matrix, for which theoretical justifications are provided. Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.

READ FULL TEXT
research
03/17/2023

Robust Mode Connectivity-Oriented Adversarial Defense: Enhancing Neural Network Robustness Against Diversified ℓ_p Attacks

Adversarial robustness is a key concept in measuring the ability of neur...
research
08/24/2023

Geodesic Mode Connectivity

Mode connectivity is a phenomenon where trained models are connected by ...
research
03/23/2018

Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization

Deep neural networks have lately shown tremendous performance in various...
research
10/25/2022

Exploring Mode Connectivity for Pre-trained Language Models

Recent years have witnessed the prevalent application of pre-trained lan...
research
02/25/2021

Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling

With a better understanding of the loss surfaces for multilayer networks...
research
10/04/2022

SAM as an Optimal Relaxation of Bayes

Sharpness-aware minimization (SAM) and related adversarial deep-learning...
research
09/05/2020

Optimizing Mode Connectivity via Neuron Alignment

The loss landscapes of deep neural networks are not well understood due ...

Please sign up or login with your details

Forgot password? Click here to reset