Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

06/08/2021
by   Julius von Kügelgen, et al.
11

Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

page 22

page 23

page 24

page 28

12/07/2021

ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints

Learning self-supervised video representation predominantly focuses on d...
10/08/2021

SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning

Self-supervised learning has been shown to be very effective in learning...
03/15/2012

Invariant Gaussian Process Latent Variable Models and Application in Causal Discovery

In nonlinear latent variable models or dynamic models, if we consider th...
11/29/2021

Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective

Learning behavioral patterns from observational data has been a de-facto...
12/08/2020

Conditional Generation of Medical Images via Disentangled Adversarial Inference

Synthetic medical image generation has a huge potential for improving he...
05/13/2022

Toward a Geometrical Understanding of Self-supervised Contrastive Learning

Self-supervised learning (SSL) is currently one of the premier technique...
10/28/2021

Self-Supervised Learning Disentangled Group Representation as Feature

A good visual representation is an inference map from observations (imag...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.