Automatic Shortcut Removal for Self-Supervised Representation Learning

02/20/2020
by   Matthias Minderer, et al.
0

In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for removing shortcut features automatically. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.

READ FULL TEXT

page 6

page 8

page 13

page 14

page 15

page 16

page 17

page 18

research
11/15/2022

Homomorphic Self-Supervised Learning

In this work, we observe that many existing self-supervised learning alg...
research
03/22/2016

Learning Representations for Automatic Colorization

We develop a fully automatic image colorization system. Our approach lev...
research
05/03/2021

Curious Representation Learning for Embodied Intelligence

Self-supervised representation learning has achieved remarkable success ...
research
11/17/2020

Can Semantic Labels Assist Self-Supervised Visual Representation Learning?

Recently, contrastive learning has largely advanced the progress of unsu...
research
08/18/2021

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Advanced self-supervised visual representation learning methods rely on ...
research
01/16/2020

Self-supervised visual feature learning with curriculum

Self-supervised learning techniques have shown their abilities to learn ...
research
01/25/2021

The emergence of visual semantics through communication games

The emergence of communication systems between agents which learn to pla...

Please sign up or login with your details

Forgot password? Click here to reset