Toward a Geometrical Understanding of Self-supervised Contrastive Learning

05/13/2022
by   Romain Cosentino, et al.
0

Self-supervised learning (SSL) is currently one of the premier techniques to create data representations that are actionable for transfer learning in the absence of human annotations. Despite their success, the underlying geometry of these representations remains elusive, which obfuscates the quest for more robust, trustworthy, and interpretable models. In particular, mainstream SSL techniques rely on a specific deep neural network architecture with two cascaded neural networks: the encoder and the projector. When used for transfer learning, the projector is discarded since empirical results show that its representation generalizes more poorly than the encoder's. In this paper, we investigate this curious phenomenon and analyze how the strength of the data augmentation policies affects the data embedding. We discover a non-trivial relation between the encoder, the projector, and the data augmentation strength: with increasingly larger augmentation policies, the projector, rather than the encoder, is more strongly driven to become invariant to the augmentations. It does so by eliminating crucial information about the data by learning to project it into a low-dimensional space, a noisy estimate of the data manifold tangent plane in the encoder representation. This analysis is substantiated through a geometrical perspective with theoretical and empirical results.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/18/2022

The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning

Self-supervised learning (SSL) has emerged as a desirable paradigm in co...
research
06/01/2023

Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation

Good data augmentation is one of the key factors that lead to the empiri...
research
06/16/2020

Visual Chirality

How can we tell whether an image has been mirrored? While we understand ...
research
12/16/2021

High Fidelity Visualization of What Your Self-Supervised Representation Knows About

Discovering what is learned by neural networks remains a challenge. In s...
research
08/31/2020

A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

Contrastive self-supervised learning (CSL) is an approach to learn usefu...
research
06/10/2022

Masked Autoencoders are Robust Data Augmentors

Deep neural networks are capable of learning powerful representations to...
research
06/25/2023

A Self-Encoder for Learning Nearest Neighbors

We present the self-encoder, a neural network trained to guess the ident...

Please sign up or login with your details

Forgot password? Click here to reset