What Do We Maximize in Self-Supervised Learning?

07/20/2022
by   Ravid Shwartz-Ziv, et al.
175

In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction. As a first step, we demonstrate how information-theoretic quantities can be obtained for a deterministic network, offering a possible alternative to prior work that relies on stochastic models. This enables us to demonstrate how VICReg can be (re)discovered from first principles and its assumptions about data distribution. Furthermore, we empirically demonstrate the validity of our assumptions, confirming our novel understanding of VICReg. Finally, we believe that the derivation and insights we obtain can be generalized to many other SSL methods, opening new avenues for theoretical and practical understanding of SSL and transfer learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/01/2023

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization

In this paper, we provide an information-theoretic perspective on Varian...
research
04/19/2023

To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review

Deep neural networks have demonstrated remarkable performance in supervi...
research
11/17/2017

Improvements to context based self-supervised learning

We develop a set of methods to improve on the results of self-supervised...
research
01/31/2022

Adversarial Masking for Self-Supervised Learning

We propose ADIOS, a masked image model (MIM) framework for self-supervis...
research
04/14/2023

Tempo vs. Pitch: understanding self-supervised tempo estimation

Self-supervision methods learn representations by solving pretext tasks ...
research
10/02/2022

What shapes the loss landscape of self-supervised learning?

Prevention of complete and dimensional collapse of representations has r...
research
10/01/2020

Understanding Self-supervised Learning with Dual Deep Networks

We propose a novel theoretical framework to understand self-supervised l...

Please sign up or login with your details

Forgot password? Click here to reset