Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations

09/29/2022
by   Grégoire Mialon, et al.
13

Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector's output. This study highlights important properties of such strategy, which we coin Variance-Covariance regularization (VCReg). More precisely, we show that VCReg enforces pairwise independence between the features of the learned representation. This result emerges by bridging VCReg applied on the projector's output to kernel independence criteria applied on the projector's input. This provides the first theoretical motivations and explanations of VCReg. We empirically validate our findings where (i) we observe that SSL methods employing VCReg learn visual representations with greater pairwise independence than other methods, (i) we put in evidence which projector's characteristics favor pairwise independence, and show it to emerge independently from learning the projector, (ii) we use these findings to obtain nontrivial performance gains for VICReg, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis. We hope that our findings will support the adoption of VCReg in SSL and beyond.

READ FULL TEXT

page 20

page 21

page 22

page 23

research
12/09/2022

Predictor networks and stop-grads provide implicit variance regularization in BYOL/SimSiam

Self-supervised learning (SSL) learns useful representations from unlabe...
research
05/11/2021

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Recent self-supervised methods for image representation learning are bas...
research
06/23/2023

Variance-Covariance Regularization Improves Representation Learning

Transfer learning has emerged as a key approach in the machine learning ...
research
05/19/2023

S-JEA: Stacked Joint Embedding Architectures for Self-Supervised Visual Representation Learning

The recent emergence of Self-Supervised Learning (SSL) as a fundamental ...
research
09/02/2021

VIbCReg: Variance-Invariance-better-Covariance Regularization for Self-Supervised Learning on Time Series

Self-supervised learning for image representations has recently had many...
research
11/13/2022

Methods for Recovering Conditional Independence Graphs: A Survey

Conditional Independence (CI) graphs are a type of probabilistic graphic...

Please sign up or login with your details

Forgot password? Click here to reset