Understanding Failure Modes of Self-Supervised Learning

03/03/2022
by   Neha Mukund Kalibhat, et al.
59

Self-supervised learning methods have shown impressive results in downstream classification tasks. However, there is limited work in understanding their failure models and interpreting the learned representations of these models. In this paper, we tackle these issues and study the representation space of self-supervised models by understanding the underlying reasons for misclassifications in a downstream task. Over several state-of-the-art self-supervised models including SimCLR, SwaV, MoCo V2 and BYOL, we observe that representations of correctly classified samples have few discriminative features with highly deviated values compared to other features. This is in a clear contrast with representations of misclassified samples. We also observe that noisy features in the representation space often correspond to spurious attributes in images making the models less interpretable. Building on these observations, we propose a sample-wise Self-Supervised Representation Quality Score (or, Q-Score) that, without access to any label information, is able to predict if a given sample is likely to be misclassified in the downstream task, achieving an AUPRC of up to 0.90. Q-Score can also be used as a regularization to remedy low-quality representations leading to 3.26 accuracy of SimCLR on ImageNet-100. Moreover, we show that Q-Score regularization increases representation sparsity, thus reducing noise and improving interpretability through gradient heatmaps.

READ FULL TEXT

page 2

page 5

page 6

page 8

page 13

page 14

page 15

page 16

research
02/01/2023

Image-Based Vehicle Classification by Synergizing Features from Supervised and Self-Supervised Learning Paradigms

This paper introduces a novel approach to leverage features learned from...
research
12/16/2021

High Fidelity Visualization of What Your Self-Supervised Representation Knows About

Discovering what is learned by neural networks remains a challenge. In s...
research
10/11/2022

Self-supervised debiasing using low rank regularization

Spurious correlations can cause strong biases in deep neural networks, i...
research
05/24/2023

Reverse Engineering Self-Supervised Learning

Self-supervised learning (SSL) is a powerful tool in machine learning, b...
research
09/07/2022

Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probing

Self-supervised visual representation learning has recently attracted si...
research
04/01/2022

Simplicial Embeddings in Self-Supervised Learning and Downstream Classification

We introduce Simplicial Embeddings (SEMs) as a way to constrain the enco...
research
01/28/2023

Deciphering the Projection Head: Representation Evaluation Self-supervised Learning

Self-supervised learning (SSL) aims to learn intrinsic features without ...

Please sign up or login with your details

Forgot password? Click here to reset