DeepAI AI Chat
Log In Sign Up

Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness

by   Tianyu Pang, et al.
Tsinghua University

Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we instead focus on inducing locally dense sample distribution, i.e., high sample density in the feature space which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants induce inappropriate sample density distributions in the feature space, which inspires us to design appropriate training objectives. Specifically, we propose the Max-Mahalanobis center (MMC) loss to create high-density regions for better robustness. It encourages the learned features to gather around the preset class centers with optimal inter-class dispersion. Comparing to the SCE loss and its variants, we empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation.


On the Separability of Classes with the Cross-Entropy Loss Function

In this paper, we focus on the separability of classes with the cross-en...

Improving Adversarial Robustness by Encouraging Discriminative Features

Deep neural networks (DNNs) have achieved state-of-the-art results in va...

Vehicle Re-identification Based on Dual Distance Center Loss

Recently, deep learning has been widely used in the field of vehicle re-...

Leveraging Class Similarity to Improve Deep Neural Network Robustness

Traditionally artificial neural networks (ANNs) are trained by minimizin...

Adaptive Verifiable Training Using Pairwise Class Similarity

Verifiable training has shown success in creating neural networks that a...

Unlabeled Data Improves Adversarial Robustness

We demonstrate, theoretically and empirically, that adversarial robustne...

Adversarial Robustness via Fisher-Rao Regularization

Adversarial robustness has become a topic of growing interest in machine...