Generalization Bounds for Adversarial Contrastive Learning

02/21/2023
by   Xin Zou, et al.
0

Deep networks are well-known to be fragile to adversarial attacks, and adversarial training is one of the most popular methods used to train a robust model. To take advantage of unlabeled data, recent works have applied adversarial training to contrastive learning (Adversarial Contrastive Learning; ACL for short) and obtain promising robust performance. However, the theory of ACL is not well understood. To fill this gap, we leverage the Rademacher complexity to analyze the generalization performance of ACL, with a particular focus on linear models and multi-layer neural networks under ℓ_p attack (p ≥ 1). Our theory shows that the average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task. The experimental results validate our theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/01/2021

ℓ_∞-Robustness and Beyond: Unleashing Efficient Adversarial Training

Neural networks are vulnerable to adversarial attacks: adding well-craft...
research
01/20/2022

Unsupervised Graph Poisoning Attack via Contrastive Loss Back-propagation

Graph contrastive learning is the state-of-the-art unsupervised graph re...
research
04/30/2023

Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization

Adversarial contrastive learning (ACL), without requiring labels, incorp...
research
11/27/2022

Adversarial Rademacher Complexity of Deep Neural Networks

Deep neural networks are vulnerable to adversarial attacks. Ideally, a r...
research
04/13/2022

Overparameterized Linear Regression under Adversarial Attacks

As machine learning models start to be used in critical applications, th...
research
08/15/2020

On the Generalization Properties of Adversarial Training

Modern machine learning and deep learning models are shown to be vulnera...
research
04/25/2023

Learning Robust Deep Equilibrium Models

Deep equilibrium (DEQ) models have emerged as a promising class of impli...

Please sign up or login with your details

Forgot password? Click here to reset