VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

05/11/2021
by   Adrien Bardes, et al.
23

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/21/2022

TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning

We present Transformation Invariance and Covariance Contrast (TiCo) for ...
research
03/01/2023

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization

In this paper, we provide an information-theoretic perspective on Varian...
research
09/02/2021

VIbCReg: Variance-Invariance-better-Covariance Regularization for Self-Supervised Learning on Time Series

Self-supervised learning for image representations has recently had many...
research
12/09/2022

Predictor networks and stop-grads provide implicit variance regularization in BYOL/SimSiam

Self-supervised learning (SSL) learns useful representations from unlabe...
research
09/29/2022

Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations

Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W...
research
06/23/2023

Variance-Covariance Regularization Improves Representation Learning

Transfer learning has emerged as a key approach in the machine learning ...
research
11/09/2022

Side-Informed Steganography for JPEG Images by Modeling Decompressed Images

Side-informed steganography has always been among the most secure approa...

Please sign up or login with your details

Forgot password? Click here to reset