DeepAI AI Chat
Log In Sign Up

Automatic Shortcut Removal for Self-Supervised Representation Learning

by   Matthias Minderer, et al.

In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for removing shortcut features automatically. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.


page 6

page 8

page 13

page 14

page 15

page 16

page 17

page 18


Homomorphic Self-Supervised Learning

In this work, we observe that many existing self-supervised learning alg...

Learning Representations for Automatic Colorization

We develop a fully automatic image colorization system. Our approach lev...

Curious Representation Learning for Embodied Intelligence

Self-supervised representation learning has achieved remarkable success ...

Can Semantic Labels Assist Self-Supervised Visual Representation Learning?

Recently, contrastive learning has largely advanced the progress of unsu...

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Advanced self-supervised visual representation learning methods rely on ...

Self-supervised visual feature learning with curriculum

Self-supervised learning techniques have shown their abilities to learn ...

The emergence of visual semantics through communication games

The emergence of communication systems between agents which learn to pla...