Understanding the Role of Adversarial Regularization in Supervised Learning

10/01/2020
by   Litu Rout, et al.
0

Despite numerous attempts sought to provide empirical evidence of adversarial regularization outperforming sole supervision, the theoretical understanding of such phenomena remains elusive. In this study, we aim to resolve whether adversarial regularization indeed performs better than sole supervision at a fundamental level. To bring this insight into fruition, we study vanishing gradient issue, asymptotic iteration complexity, gradient flow and provable convergence in the context of sole supervision and adversarial regularization. The key ingredient is a theoretical justification supported by empirical evidence of adversarial acceleration in gradient descent. In addition, motivated by a recently introduced unit-wise capacity based generalization bound, we analyze the generalization error in adversarial framework. Guided by our observation, we cast doubts on the ability of this measure to explain generalization. We therefore leave as open questions to explore new measures that can explain generalization behavior in adversarial learning. Furthermore, we observe an intriguing phenomenon in the neural embedded vector space while contrasting adversarial learning with sole supervision.

READ FULL TEXT

page 17

page 18

page 19

page 20

research
10/30/2019

Is Supervised Learning With Adversarial Features Provably Better Than Sole Supervision?

Generative Adversarial Networks (GAN) have shown promising results on a ...
research
06/11/2021

Towards Understanding Generalization via Decomposing Excess Risk Dynamics

Generalization is one of the critical issues in machine learning. Howeve...
research
10/06/2018

Over-parameterization Improves Generalization in the XOR Detection Problem

Empirical evidence suggests that neural networks with ReLU activations g...
research
02/13/2018

Towards Understanding the Generalization Bias of Two Layer Convolutional Linear Classifiers with Gradient Descent

A major challenge in understanding the generalization of deep learning i...
research
10/01/2019

Robust Semantic Parsing with Adversarial Learning for Domain Generalization

This paper addresses the issue of generalization for Semantic Parsing in...
research
10/01/2020

Why Adversarial Interaction Creates Non-Homogeneous Patterns: A Pseudo-Reaction-Diffusion Model for Turing Instability

Long after Turing's seminal Reaction-Diffusion (RD) model, the elegance ...
research
05/07/2021

Uniform Convergence, Adversarial Spheres and a Simple Remedy

Previous work has cast doubt on the general framework of uniform converg...

Please sign up or login with your details

Forgot password? Click here to reset