Posterior Differential Regularization with f-divergence for Improving Model Robustness

by   Hao Cheng, et al.

We address the problem of enhancing model robustness through regularization. Specifically, we focus on methods that regularize the model posterior difference between clean and noisy inputs. Theoretically, we provide a connection of two recent methods, Jacobian Regularization and Virtual Adversarial Training, under this framework. Additionally, we generalize the posterior differential regularization to the family of f-divergences and characterize the overall regularization framework in terms of Jacobian matrix. Empirically, we systematically compare those regularizations and standard BERT training on a diverse set of tasks to provide a comprehensive profile of their effect on model in-domain and out-of-domain generalization. For both fully supervised and semi-supervised settings, our experiments show that regularizing the posterior differential with f-divergence can result in well-improved model robustness. In particular, with a proper f-divergence, a BERT-base model can achieve comparable generalization as its BERT-large counterpart for in-domain, adversarial and domain shift scenarios, indicating the great potential of the proposed framework for boosting model generalization for NLP models.


page 1

page 2

page 3

page 4


Adaptive Regularization for Adversarial Training

Adversarial training, which is to enhance robustness against adversarial...

Distributional Smoothing with Virtual Adversarial Training

We propose local distributional smoothness (LDS), a new notion of smooth...

Adversarial Transformations for Semi-Supervised Learning

We propose a Regularization framework based on Adversarial Transformatio...

Towards Improving Adversarial Training of NLP Models

Adversarial training, a method for learning robust deep neural networks,...

Label-Efficient Domain Generalization via Collaborative Exploration and Generalization

Considerable progress has been made in domain generalization (DG) which ...

Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization

Machine learning (ML) robustness and domain generalization are fundament...

NoiLIn: Do Noisy Labels Always Hurt Adversarial Training?

Adversarial training (AT) based on minimax optimization is a popular lea...