Is Self-Supervised Learning More Robust Than Supervised Learning?

06/10/2022
by   Yuanyi Zhong, et al.
8

Self-supervised contrastive learning is a powerful tool to learn visual representation without labels. Prior work has primarily focused on evaluating the recognition accuracy of various pre-training algorithms, but has overlooked other behavioral aspects. In addition to accuracy, distributional robustness plays a critical role in the reliability of machine learning models. We design and conduct a series of robustness tests to quantify the behavioral differences between contrastive learning and supervised learning to downstream or pre-training data distribution changes. These tests leverage data corruptions at multiple levels, ranging from pixel-level gamma distortion to patch-level shuffling and to dataset-level distribution shift. Our tests unveil intriguing robustness behaviors of contrastive and supervised learning. On the one hand, under downstream corruptions, we generally observe that contrastive learning is surprisingly more robust than supervised learning. On the other hand, under pre-training corruptions, we find contrastive learning vulnerable to patch shuffling and pixel intensity change, yet less sensitive to dataset-level distribution change. We attempt to explain these results through the role of data augmentation and feature space properties. Our insight has implications in improving the downstream robustness of supervised learning.

READ FULL TEXT

page 1

page 15

page 16

research
06/18/2020

Recovering Petaflops in Contrastive Semi-Supervised Learning of Visual Representations

We investigate a strategy for improving the computational efficiency of ...
research
11/04/2021

Leveraging Time Irreversibility with Order-Contrastive Pre-training

Label-scarce, high-dimensional domains such as healthcare present a chal...
research
05/01/2023

CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations

Geo-tagged images are publicly available in large quantities, whereas la...
research
05/17/2021

Divide and Contrast: Self-supervised Learning from Uncurated Data

Self-supervised learning holds promise in leveraging large amounts of un...
research
05/15/2023

Improved baselines for vision-language pre-training

Contrastive learning has emerged as an efficient framework to learn mult...
research
06/03/2023

Uncovering the Hidden Dynamics of Video Self-supervised Learning under Distribution Shifts

Video self-supervised learning (VSSL) has made significant progress in r...
research
11/06/2021

Towards noise robust trigger-word detection with contrastive learning pre-task for fast on-boarding of new trigger-words

Trigger-word detection plays an important role as the entry point of use...

Please sign up or login with your details

Forgot password? Click here to reset