An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization

03/01/2023
by   Ravid Shwartz-Ziv, et al.
0

In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning. To do so, we first demonstrate how information-theoretic quantities can be obtained for deterministic networks as an alternative to the commonly used unrealistic stochastic networks assumption. Next, we relate the VICReg objective to mutual information maximization and use it to highlight the underlying assumptions of the objective. Based on this relationship, we derive a generalization bound for VICReg, providing generalization guarantees for downstream supervised learning tasks and present new self-supervised learning methods, derived from a mutual information maximization objective, that outperform existing methods in terms of performance. This work provides a new information-theoretic perspective on self-supervised learning and Variance-Invariance-Covariance Regularization in particular and guides the way for improved transfer learning via information-theoretic self-supervised learning objectives.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/19/2023

To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review

Deep neural networks have demonstrated remarkable performance in supervi...
research
07/20/2022

What Do We Maximize in Self-Supervised Learning?

In this paper, we examine self-supervised learning methods, particularly...
research
09/16/2022

Self-Supervised Learning with an Information Maximization Criterion

Self-supervised learning allows AI systems to learn effective representa...
research
06/23/2023

Variance-Covariance Regularization Improves Representation Learning

Transfer learning has emerged as a key approach in the machine learning ...
research
05/11/2021

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Recent self-supervised methods for image representation learning are bas...
research
06/15/2021

Self-Supervised Learning with Kernel Dependence Maximization

We approach self-supervised learning of image representations from a sta...
research
10/18/2019

A Mutual Information Maximization Perspective of Language Representation Learning

We show state-of-the-art word representation learning methods maximize a...

Please sign up or login with your details

Forgot password? Click here to reset