Demystifying Self-Supervised Learning: An Information-Theoretical Framework

1 Introduction

Self-supervised learning (SSL) zhang2016colorful ; srivastava2015unsupervised ; devlin2018bert ; oord2018representation ; bachman2019learning ; chen2020simple ; tian2019contrastive ; hjelm2018learning ; he2019momentum ; kong2019mutual ; arora2019theoretical learns representations using a proxy objective (i.e., SSL objective) between inputs and self-defined signals. Empirical evidences suggest that the learned representations can generalize well to a wide range of downstream tasks, even when there is no clear connection between the SSL objective and the downstream tasks. For example, BERT devlin2018bert

defines a prediction loss (i.e., a SSL objective) from non-masked words (i.e., inputs) to masked words (i.e., self-supervised signals). Then, one takes BERT as word features extractor and adopts the word features to various natural language processing applications, spanning sentiment analysis, question answering, dialogue system, and named-entity recognition

young2018recent . Despite showing success in practice, there are only a few work arora2019theoretical providing theoretical insights into SSL. In particular, Arora et al. arora2019theoretical presented provable guarantees on the performance for downstream classification task when using contrastive learning objectives in SSL. Our work shares a similar goal of demystifying SSL, but approaching it from an Information Theory cover2012elements perspective to understand when and why self-supervised learning is likely to work.

In this paper, we argue that a good representation learning procedure is the one that learns representations that are maximally compressed and include only the information required for the downstream tasks. In other words, the representations should maximally extract task-relevant and discard task-irrelevant information. To connect these compressed representation learning procedure and SSL (which has no access to downstream tasks), we rely on a core assumption: only the shared information between the input and self-supervised signals contributes to the downstream tasks. To see that this assumption is likely hold in practice, we again take BERT devlin2018bert as an example. In BERT devlin2018bert , the information shared across masked and non-masked words is referred to as contextual information. Our assumption states that the contextual information contributes to the downstream tasks and not the exclusive information in masked words or non-masked words. Another example is visual representation learning in SimCLR chen2020simple , where the authors apply different image augmentations on a given image, treating one of them as input and the other one as the corresponding self-supervised signal. Our assumption states that only the shared information (i.e., the content of the image) between the augmented images contributes to the downstream tasks, which is in accord that the image augmentations (e.g., changing the style of an image) should not affect the labels of images.

Based on this assumption, we develop an unsupervised compressed representation learning strategy. In particular, we extract task-relevant information by maximizing the mutual information between the learned representations and the self-supervised signals. Then, we discard task-irrelevant information by minimizing the conditional entropy of the learned representations given the self-supervised signals. We show this strategy 1) includes prior arts for SSL on contrastive agrawal2015learning ; arandjelovic2017look ; jayaraman2015learning ; oord2018representation ; bachman2019learning ; chen2020simple ; tian2019contrastive ; hjelm2018learning ; he2019momentum ; kong2019mutual ; ozair2019wasserstein ; arora2019theoretical ; henaff2019data and predictive learning zhang2016colorful ; pathak2016context ; vondrick2016generating ; tulyakov2018mocogan ; srivastava2015unsupervised ; peters2018deep ; devlin2018bert ; dai2019transformer ; bai2018empirical approaches; 2) paves the way to a larger space of composing SSL objectives; and 3) leads us a discussion on limitations and challenges of using these objectives. For instance, we can combine both contrastive and predictive learning approaches as our SSL objective, being aware that the contrastive objective requires larger batch size and the predictive objective is hard to optimize if the self-supervised signals are high-dimensional.

We first conduct controlled experiments on visual representation learning to 1) verify that the self-supervisedly learned representation could extract task-relevant and discard task-irrelevant information; and 2) compare different compositions of SSL objectives. Then, we perform self-supervised visual-textual representation learning in a challenging setting that input and self-supervised signals lie in very different modalities. We make our experiments publicly available at https://github.com/yaohungt/Demystifying_Self_Supervised_Learning.

2 An Information-Theoretical Framework for Self-supervised Learning

In this section we aim to show self-supervised learning (SSL) can learn a representation that is beneficial for downstream tasks. For the input, we denote its random variable as

X

, sample space as

X

, and outcome as

x

. Similarly, for the self-supervised signal, we denote its random variable/ sample space/ outcome as

S

S

s

. Two sample spaces can be different:

X \neq S

. We learn a representation (

Z_{X}

Z

z_{x}

) from the input through a deterministic mapping

F_{X}

Z_{X} = F_{X} (X)

. The information required for downstream tasks is referred to as “task-relevant information”:

T

T

t

. Note that SSL has no access to the task-relevant information. Lastly, we use

I (A; B)

to represent mutual information,

I (A; B | C)

to represent conditional mutual information, and

H (A | B)

to represent conditional entropy for random variables

A

B

C

. We provide high-level takeaways for our main results in Figure 1.

Figure 1: High-level takeaways for our main results using information diagrams. (a) For self-supervised learning, we show that minimizing $H (Z_{X} | S)$ acts to discard task-irrelevant information and maximizing $I (Z_{X}; S)$ acts to extract task-relevant information, even when these two objectives have no access to the downstream tasks. (b) The resulting learned representation ${Z_{X}}^{*}$ contains only and no more than the shared information between $X$ / $S$ . We demonstrate that ${Z_{X}}^{*}$ extracts all task-relevant information from $X / S$ and $I (X; S | T)$ is the information that cannot be discarded. (c) Our derivations are based on a core assumption: the input and self-supervised signals are mutually redundant for the downstream tasks. The assumption suggests the exclusive information in input and self-supervised signal is what we can discard.

2.1 Redundancy Assumption and Determinism

The derivations throughout the paper rely on the following redundancy assumption and determinism lemma. First, we assume redundancy between the input $X$ and self-supervised signal $S$ :

Assumption 1 (Redundancy).

The input is redundant to the self-supervised signal for the task-relevant information. In other words, we assume the following conditional independence: $T ⊥ ⊥ S | X$ or equivalently $I (T; S | X) = 0$ . We assume the redundancy also holds when we swap $X$ and $S$ , and hence $T ⊥ ⊥ X | S$ or equivalently $I (T; X | S) = 0$ . By mutual redundancy, $I (S; T) = I (X; T) = I (S, X; T)$ .

Assumption 1 states that the information required for the downstream tasks lies only in the shared information between the input and self-supervised signals. We provide an intuition by relating the assumption to Multiview learning xu2013survey ; sridharan2008information . Multiview learning extracts representations from data across different views, and it assumes each view provides the same task-relevant information. In SSL, we can regard the input and self-supervised signals as different views of the data. For instance, in contrastive visual representation learning hjelm2018learning ; chen2020simple , the input and the corresponding self-supervised signal are the same image with different image augmentations (images with different views).

Next, we provide a useful lemma using the fact that $F_{X}$ is a deterministic mapping:

Lemma 1 (Determinism).

If $P (Z_{X} | X)$ is Dirac, then the following conditional independence holds: $T ⊥ ⊥ Z_{X} | X$ and $S ⊥ ⊥ Z_{X} | X$

, given by a Markov chain

S \leftrightarrow T \leftrightarrow X \to Z_{X}

¹¹1The Markov chain is naturally satisfied when

F_{X}

is a deterministic mapping. If

F_{X}

is random, the Markov chain needs to be further assumed to satisfy the conditional independence:

T ⊥ ⊥ Z_{X} | X

and

S ⊥ ⊥ Z_{X} | X

This lemma simply states that $Z_{X}$ contains no more information than $X$ .

2.2 Supervised Representation Learning

Under a supervised setting, to learn representations which contain only and no more than the information required for the downstream tasks, we consider the following objectives:

Definition 1 (Supervised Representation Learning).

Uncompressed and compressed supervised representation are defined as

Z_{X}^{s u p} = a r g m a x Z_{X} I (Z_{X}; T) a n d Z_{X}^{s u p_{c o m}} = a r g m i n Z_{X} H (Z_{X} | T) s . t . I (Z_{X}; T) i s m a x i m i z e d .

Then, $I (Z_{X}^{s u p}; T) = I (Z_{X}^{s u p_{c o m}}; T) = I (S, X; T)$ contains all task-relevant information .

Proof.

Adopting Data Processing Inequality cover2012elements in the Markov chain $S \leftrightarrow T \leftrightarrow X \to Z_{X}$ (Lemma 1), $I (Z_{X}; T)$ is maximized at $I (X; T)$ . $I (X; T) = I (S, X; T)$ by Assumption 1. ∎

The definition shows the supervisedly learned representation $Z_{X}^{s u p}$ / $Z_{X}^{s u p_{c o m}}$ can extract relevant information for the downstream tasks. Next, we provide a justification that minimizing $H (Z_{X} | T)$ ²²2To discard task-irrelevant information, an alternative objective is minimizing $I (Z_{X}; X, S | T)$ , which represents the information between $Z_{X}$ and $X / S$ that are irrelevant to $T$ . However, minimizing the conditional mutual information (i.e., $I (Z_{X}; X, S | T)$ ) requires a min-max optimization, which may cause instability in practice. Hence, we consider minimizing $H (Z_{X} | T)$ , which does not contain a min-max optimization. leads to compressed representations. Minimizing $H (Z_{X} | T)$ reduces the randomness from $T$ to $Z_{X}$ , and the randomness is regarded as the incompressibility calude2013information . Hence, when satisfying the constraint “ $I (Z_{X}; T)$ is maximized”, minimizing $H (Z_{X} | T)$ leads to a more compressed representation (discarding superfluous information). Note that our analysis does not constrain the type of $T$ , which can be classification, regression, or clustering.

2.3 A Self-supervised Representation Learning Strategy

In Definition 1, we discuss uncompressed and compressed supervised representation learning objectives. To bridge the gap between supervised and self-supervised learning, we perform the following supervision decomposition (from the downstream tasks to the self-supervised signals):

Lemma 2 (Supervision Decomposition).

We consider the supervision decomposition from $T$ to $S$ :

I (Z_{X}; S) = I (Z_{X}; T) + I (Z_{X}; S | T) a n d H (Z_{X} | S) = H (Z_{X} | T) - I (Z_{X}; S | T) .

Also, $I (X; S) = I (X; T) + I (X; S | T) a n d H (X | S) = H (X | T) - I (X; S | T) .$

The decomposition allows us to 1) perform supervision on $S$ (i.e., self-supervised learning) instead of $T$ (i.e., supervised learning); 2) associate supervisedly- and self-supervisedly-learned representations; and 3) characterize the compression gap from supervised to self-supervised learning. Formally,

Definition 2 (Self-supervised Representation Learning).

Uncompressed and compressed self-supervised representation are defined as

Z_{X}^{s s l} = a r g m a x Z_{X} I (Z_{X}; S) a n d Z_{X}^{s s l_{c o m}} = a r g m i n Z_{X} H (Z_{X} | S) s . t . I (Z_{X}; S) i s m a x i m i z e d .

Then, $I (Z_{X}^{s s l}; S) = I (Z_{X}^{s s l_{c o m}}; S) = I (X; S)$ contains all the shared information between $X$ and $S$ .

Proof.

Adopting Data Processing Inequality cover2012elements in the Markov chain $S \leftrightarrow T \leftrightarrow X \to Z_{X}$ (Lemma 1), $I (Z_{X}; S)$ is maximized at $I (X; S)$ . ∎

Theorem 1 (Inclusion).

Uncompressed and compressed self-supervised representation extract all task-relevant information, suggesting $I (Z_{X}^{s s l}; T) = I (Z_{X}^{s s l_{c o m}}; T) = I (Z_{X}^{s u p}; T) = I (X, S; T)$ :

\begin{matrix} I f I (Z_{X}; S) i s m a x i m i z e d t h e n I (Z_{X}; T) i s m a x i m i z e d a n d I (Z_{X}; S | T) i s m a x i m i z e d . \end{matrix}

In other words, compressed self-supervised representation is a subset of uncompressed self-supervised representation, and the later one is a subset of supervised representation: $Z_{X}^{s s l_{c o m}} \subseteq Z_{X}^{s s l} \subseteq Z_{X}^{s u p}$ .

Proof.

Adopting Data Processing Inequality cover2012elements in $S \leftrightarrow T \leftrightarrow X \to Z_{X}$ (Lemma 1), $I (Z_{X}; S | T)$ is maximized at $I (X; S | T)$ . Then, bringing the results in Definitions 1 and 2 into Lemma 2, we conclude $I (Z_{X}; S)$ is maximized if and only if $I (Z_{X}; T)$ and $I (Z_{X}; S | T)$ are both maximized. ∎

Theorem 2 (Compression Gap).

Compressed self-supervised representation cannot discard all task-irrelevant information, where a compression gap $I (X; S | T)$ exists:

\begin{matrix} Z_{X}^{s s l_{c o m}} & = a r g m i n Z_{X} H (Z_{X} | S) s . t . I (Z_{X}; S) i s m a x i m i z e d = a r g m i n Z_{X} H (Z_{X} | T) s . t . I (Z_{X}; T) i s m a x i m i z e d a n d I (Z_{X}; S | T) i s m a x i m i z e d \end{matrix}

with $I (Z_{X}^{s s l_{c o m}}; S | T) = I (X; S | T)$ is the information that cannot be discarded in SSL.

Proof.

In Theorem 1, we show that $I (Z_{X}; S)$ is maximized if and only if $I (Z_{X}; T)$ and $I (Z_{X}; S | T)$ are both maximized, where $I (Z_{X}; S | T)$ is maximized at $I (X; S | T)$ . Following Lemma 2, $H (Z_{X} | S) = H (Z_{X} | T) - I (Z_{X}; S | T) = H (Z_{X} | T) - I (X; S | T)$ , where $I (X; S | T)$ is constant w.r.t. $Z_{X}$ . We conclude the proof by plugging-in the result into Definition 2. ∎

As a summary, Definition 2 defines our compressed SSL strategy. Theorem 1 indicates that this strategy can extract as much task-relevant information as the supervised learned one. For how much task-irrelevant information can be discarded, Theorem 2 indicates a compression gap between the supervised and the self-supervised learning.

2.4 Relations with Contrastive and Predictive Representation Learning

We now associate our self-supervised representation learning strategy (Definition 2) with prior SSL objectives, especially for contrastive agrawal2015learning ; arandjelovic2017look ; jayaraman2015learning ; oord2018representation ; bachman2019learning ; chen2020simple ; tian2019contrastive ; hjelm2018learning ; he2019momentum ; kong2019mutual ; ozair2019wasserstein ; arora2019theoretical ; henaff2019data and predictive zhang2016colorful ; pathak2016context ; vondrick2016generating ; tulyakov2018mocogan ; srivastava2015unsupervised ; peters2018deep ; devlin2018bert ; dai2019transformer ; bai2018empirical learning objectives. We illustrate important remarks in Figure 2.

Figure 2: Remarks on contrastive and predictive learning objectives for self-supervised learning. Between the representation $Z_{X}$ and the self-supervised signal $S$ , contrastive objective performs mutual information maximization and predictive objectives perform log conditional likelihood maximization. We show that the SSL objectives aim at extracting task-relevant and discarding task-irrelevant information. Last, we summarize the computational blocks for practical deployments for these objectives.

Contrastive Learning

We define the contrastive learning objective as maximizing the mutual information $I (Z_{X}; S)$ between the learned representation $Z_{X}$ and the self-supervised signal $S$ , which maximizes dependency/contrastiveness between $Z_{X}$ and $S$ . Given Theorem 1, we have:

Corollary 1 (Contrastive learning optimally extracting task-relevant info).

If ${Z_{X}}^{*} = {a r g m a x}_{Z_{X}} I (Z_{X}; S)$ , then $I ({Z_{X}}^{*}; T) = I (X, S; T)$ contains all task-relevant information.

The corollary suggests, even having no access to the downstream tasks, maximizing $I (Z_{X}; S)$ results in $Z_{X}$ containing all the information required for the downstream tasks from $X$ / $S$ . To deploy the contrastive learning objective, recent methods propose to maximize lower bounds of mutual information belghazi2018mine ; oord2018representation ; poole2019variational ; song2019understanding or its variants such as JS-divergence poole2019variational ; hjelm2018learning between the joint density and the product of the marginal density. We denote these methods as ${m a x}_{Z_{X}, θ} I_{θ} (Z_{X}; S)$ with $θ$ representing the parameters when computing $I_{θ} (\cdot; \cdot)$ . In this work, we suggest contrastive predictive coding (CPC) oord2018representation ; tian2019contrastive

, which is a mutual information lower bound with lower variance

poole2019variational ; song2019understanding :

L_{C L} := m a x Z_{S} = F_{S} (S), Z_{X} = F_{X} (X), G E_{({z_{s}}_{1}, {z_{x}}_{1}), \dots, ({z_{s}}_{n}, {z_{x}}_{n}) \sim P^{n} (Z_{S}, Z_{X})} [\frac{1}{n} n \sum i = 1 l o g \frac{e^{⟨ G ({z_{x}}_{i}), G ({z_{s}}_{i}) ⟩}}{\frac{1}{n} \sum_{j = 1}^{n} e^{⟨ G ({z_{x}}_{i}), G ({z_{s}}_{j}) ⟩}}],

(1)

where $F_{S} : S \to Z$ is a deterministic mapping and $G$ is a project head that projects a representation in $Z$

into a lower-dimensional vector. If the input and self-supervised signals share the same sample space, i.e.,

X = S

, we can impose

F_{X} = F_{S}

(e.g., self-supervised visual representation learning chen2020simple ). The projection head,

G

, can be an identity, a linear, or a non-linear mapping. Last, we note that modeling eq. (1) or other contrastive learning objectives belghazi2018mine ; poole2019variational often require large batch size (e.g.,

n

in eq. (1)) hjelm2018learning ; he2019momentum ; chen2020simple to ensure both low variance and bias (w.r.t. the true

I (Z_{X}; S)

). Empirical work tschannen2019mutual has suggested that large variance in contrastive learning objectives may lead to worsen performance for the downstream tasks.

Forward Predictive Learning

We define the forward predictive learning as maximizing the log conditional likelihood $E_{P_{S, Z_{X}}} [l o g P (S | Z_{X})]$ from the learned representation $Z_{X}$ to the self-supervised signal $S$ , which encourages $Z_{X}$ to reconstruct $S$

. By the chain rule,

I (Z_{X}; S) = H (S) - H (S | Z_{X})

, where

H (S)

is irrelevant to

Z_{X}

. Hence, maximizing

I (Z_{X}; S)

is equivalent to maximizing

- H (S | Z_{X}) = E_{P_{S, Z_{X}}} [l o g P (S | Z_{X})]

. Given Theorem 1, we have:

Corollary 2 (Forward Predictive learning optimally extracting task-relevant info).

If ${Z_{X}}^{*} = {a r g m a x}_{Z_{X}} E_{P_{S, Z_{X}}} [l o g P (S | Z_{X})]$ , then $I ({Z_{X}}^{*}; T) = I (X, S; T)$ contains all task-relevant information.

The corollary suggests, if $z_{x}$ can perfectly reconstruct $s$ for any $(s, z_{x}) \sim P_{S, Z_{X}}$ , then $Z_{X}$ contains all the information required for the downstream tasks from $X$ / $S$ . A common approach to avoid intractability in Corollary 2 is assuming a variational distribution $Q_{ϕ} (S | Z_{X})$ with $ϕ$ representing the parameters when computing $Q_{ϕ} (\cdot | \cdot)$ . Now, we re-arrange $E_{P_{S, Z_{X}}} [l o g P (S | Z_{X})] = m a x Q_{ϕ} E_{P_{S, Z_{X}}} [l o g Q_{ϕ} (S | Z_{X})] + K L (P (S | Z_{X}) / / Q_{ϕ} (S | Z_{X})) \geq m a x Q_{ϕ} E_{P_{S, Z_{X}}} [l o g Q_{ϕ} (S | Z_{X})]$ . Hence, $E_{P_{S, Z_{X}}} [l o g Q_{ϕ} (S | Z_{X})]$ is a lower bound of $E_{P_{S, Z_{X}}} [l o g P (S | Z_{X})]$ . The bound is tight when $P (S | Z_{X}) = Q_{ϕ} (S | Z_{X})$ . $Q_{ϕ} (\cdot | \cdot)$ can be any distribution such as Gaussian or Laplacian and $ϕ$

can be a linear model, a kernel method, or a neural network. For example, MocoGAN

tulyakov2018mocogan assumes

Q

is Laplacian (i.e.,

ℓ_{1}

reconstruction loss) and

ϕ

is a deconvolutional network long2015fully . Transformer-XL dai2019transformer assumes

Q

is a categorical distribution (i.e., cross entropy loss) and

ϕ

is a Transformer network

vaswani2017attention . If we let

Q_{ϕ} (S | Z_{X})

be Gaussian

N (S | R (Z_{X}), I)

with

I

as an identity matrix, the objective becomes:

L_{F P} := m a x Z_{X} = F_{X} (X), R E_{s, z_{x} \sim P_{S, Z_{X}}} [- ∥ s - R (z_{x}) ∥_{2}^{2}],

(2)

where $R : Z \to S$ is a deterministic mapping to reconstruct $S$ from $Z$

. Note that we ignore the constants derived from the Gaussian distribution. Last, in most real-world applications, the self-supervised signal

S

has a much higher dimension than the representation

Z_{X}

. Hence, modeling a conditional generative model

Q_{ϕ} (S | Z_{X})

will be challenging. For example, considering

S

224 \times 224 \times 3

image and

Z_{X}

64 -

dimensional vector.

Inverse Predictive Learning

We define the inverse predictive learning as maximizing the log conditional likelihood $E_{P_{S, Z_{X}}} [l o g P (Z_{X} | S)]$ from the self-supervised signal $S$ to the learned representation $Z_{X}$ , which encourages $S$ to reconstruct $Z_{X}$ . Given Theorem 2 together with $- H (Z_{X} | S) = E_{P_{S, Z_{X}}} [l o g P (Z_{X} | S)]$ , we have:

Corollary 3 (Inverse Predictive learning sub-optimally discarding task-irrelevant info).

Suppose ${Z_{X}}^{*} = {a r g m a x}_{Z_{X}} E_{P_{S, Z_{X}}} [l o g P (Z_{X} | S)] s . t . I (Z_{X}; S) i s m a x i m i z e d$ . Then, ${Z_{X}}^{*}$ discards all the information, excluding $I (X; S | T)$ , irrelevant for the downstream tasks.

The corollary suggests, if $s$ can perfectly reconstruct $z_{x}$ for any $(s, z_{x}) \sim P_{S, Z_{X}}$ under the constraint that $I (Z_{X}; S)$ is maximized, then $Z_{X}$ discards the information, excluding $I (X; S | T)$ , irrelevant for the downstream tasks. Similar to the forward predictive learning, we use $E_{P_{S, Z_{X}}} [l o g Q_{ϕ} (Z_{X} | S)]$ as a lower bound of $E_{P_{S, Z_{X}}} [l o g P (Z_{X} | S)]$ . In our deployment, we take the advantage of the design in eq. (1) and let $Q_{ϕ} (Z_{X} | S)$ be Gaussian $N (Z_{X} | F_{S} (S), I)$ with $I$ being an identity matrix:

L_{I P} := m a x Z_{S} = F_{S} (S), Z_{X} = F_{X} (X) E_{z_{s}, z_{x} \sim P_{Z_{S}, Z_{X}}} [- ∥ z_{x} - z_{s} ∥_{2}^{2}] .

(3)

Note that optimizing eq. (3) alone results in a degenerated solution, e.g., learning $Z_{X}$ and $Z_{S}$ to be the same constant. As suggested in Corollary 3, we consider a constrained optimization instead of an unconstrained one.

Composing Self-supervised Learning Objectives

We have connected the SSL strategy presented in Definition 2 to contrastive learning objective in Corollary 1 and predictive learning objectives in Corollaries 2 and 3. Bringing their practical aspects together (eq. (1), (2), and (3)), we can pave the way to a larger space of composing SSL objectives:

L_{S S L} = λ_{C L} L_{C L} + λ_{F P} L_{F P} + λ_{I P} L_{I P},

(4)

where $λ_{C L}$ , $λ_{F P}$ , and $λ_{I P}$ are hyper-parameters.

3 Controlled Experiments

Visual Representation Learning

Our goal is to construct a set of controlled experiments that satisfy Assumption 1 and could empirically support Theorem 1 and 2.

$▹$ Experimental Setup. We use Omniglot dataset lake2015human in the experiments. The training set contains images from $964$ characters, and the test set contains $659$ characters. There are no characters overlap between the training and test set. Each character contains twenty examples drawn from twenty different people. We regard image as input ( $X$ ) and generate self-supervised signal ( $S$ ) by first sampling an image from the same character as the input image and then applying translation/ rotation to it. Furthermore, we represent task-relevant information ( $T$ ) by one-hot label encoding. Under this self-supervised signal construction, the exclusive information in $X$ or $S$ are drawing styles (i.e., by different people) and image augmentations, and only their shared information contribute to $T$ . To formally show the later, if $T$ representing the label for $X$ / $S$ , then $P (T | X)$ and $P (T | S)$ are Dirac. Hence, $T ⊥ ⊥ S | X$ and $T ⊥ ⊥ X | S$ , satisfying Assumption 1.

We train the feature mapping $F_{X} (\cdot)$ with SSL objectives (see eq. (4)), set $F_{S} (\cdot) = F_{X} (\cdot)$ , let $R (\cdot)$ to be symmetrical to $F_{X} (\cdot)$ , and have $G (\cdot)$ to be an identity mapping. On the test set, we fix the mapping and randomly select $5$

examples per character as the labeled examples. Then, we classify the rest of the examples using the 1-nearest neighbor classifier based on feature (i.e.,

Z_{X} = F_{X} (X)

) cosine similarity. The random performance on this task stands at

\frac{1}{659} \approx 0.15 %

. One may refer to Supplementary for more details.

Figure 3: Estimated $I (Z_{X}; T)$ / $I (Z_{X}; S)$ / $H (Z_{X} | T)$ / $H (Z_{X} | S)$ for $Z_{X}$ during self-supervised visual representation training. We estimate $I (\cdot; \cdot)$ using SMILE song2019understanding method and estimate $H (Z_{X} | \cdot)$ by its upper bound ${m i n}_{Q} - E_{P_{\cdot, Z_{X}}} [l o g Q (Z_{X} | \cdot)]$ with variational distribution $Q (Z_{X} | \cdot)$ being Gaussian $N (Z_{X} | μ (\cdot), I)$ and $μ (\cdot)$ being a learnable function ${S, T} \to Z$ .

$▹$ Results & Discussions. In Figure 3, we provide empirical analysis to support Theorem 1 and 2. We report $I (Z_{X}; T)$ / $I (Z_{X}; S)$ / $H (Z_{X} | T)$ / $H (Z_{X} | S)$ for $Z_{X}$ during training and report $I (X; T)$ / $I (X; S)$ as the upper bound of $I (Z_{X}; T)$ / $I (Z_{X}; S)$ . For the objectives, we consider $L_{C L}$ (contrastive learning only) for Theorem 1/ Corollary 1 and $L_{C L} + L_{I P}$ (contrastive and inverse predictive learning) for Theorem 2/ Corollary 3. In Figure 3 (a) and (b), we observe a positive correlation between $I (Z_{X}; S)$ and $I (Z_{X}; T)$ . Hence, it implies the SSL objectives can extract task-relevant information. Moreover, comparing to $L_{C L}$ only, $L_{C L} + L_{I P}$ has larger $I (Z_{X}; S)$

values given the same epoch or the same

I (Z_{X}; T)

. This result indicates

L_{I P}

can facilitate the representation to extract information from the self-supervised signal (

S

). Figure 3 (c) suggests positive correlation between

H (Z_{X} | T)

and

H (Z_{X} | S)

. Figure 3 (d) suggests

H (Z_{X} | S)

tends to converge after

500

epochs of training. Note that

H (Z_{X} | S)

can be regarded as the incompressibility calude2013information of

Z_{X}

given

S

. Comparing to

L_{C L}

only,

L_{C L} + L_{I P}

has smaller

H (Z_{X} | S)

values given the same number of epochs or the same

H (Z_{X} | T)

. This result implies

L_{I P}

can facilitate the representation to be more compressed.

Figure 4:
Comparisons for different compositions of SSL objectives on self-supervised visual representation training. We report mean and its standard error from
$5$ random trials.

In Figure 4, we evaluate the generalization ability on the test set for different SSL objectives. Figure 4 (a)/(b) suggest that, comparing to $L_{F P}$ , $L_{C L}$ 1) reaches better test accuracy; 2) requires shorter training epochs to reach the best performance; and 3) suffers from overfitting with long-epoch training. Combining both of them ( $L_{C L} + 0.005 L_{F P}$ ) brings their advantages together. We also find that adding $L_{I P}$ in the objective can boost model performance. According to Theorem 2 and Corollary 3, the improved performance suggests a more compressed representation results in better performance for the downstream tasks. Nonetheless, in Figure 4 (c), we find the performance is sensitive to the hyper-parameter $λ_{I P}$ for combining $L_{I P}$ . We would also like to examine whether combining $L_{C L}$ and $L_{I P}$ together can lead to improved performance in SOTA SSL framework. In Figure 4 (d), we provide experiment with SimCLR chen2020simple on CIFAR10 krizhevsky2009learning , where $λ_{I P} = 0$ refers to the exact same setup as in SimCLR (which considers only $L_{C L}$ ). By considering $L_{C L} + λ_{I P} L_{I P}$ in SimCLR, when changing $λ_{I P}$ , we observe a similar trend with our Omniglot experiment.

Visual-Textual Representation Learning

So far, we have provided empirical support for Theorem 1/ 2 and compared different SSL objectives on the visual representation learning task. Under this task, the input and self-supervised signals lie in the same domain and have the same content (i.e., images of the same character) but different styles and image augmentation. We now consider having the input and self-supervised signals lie in very different modalities - vision and text.

$▹$ Experimental Setup. We provide experiments using Microsoft COCO (MS COCO) dataset coco that contains $328$ k multi-labeled images with $2.5$ million labeled instances from $91$ objects. Each image has $5$ annotated captions describing the relationships between objects in the scenes.

We regard image as input ( $X$ ) and its textual descriptions as self-supervised signal ( $S$ ), and we use $L_{C L}$ (+ $λ_{I P} L_{I P}$ ) as our SSL objective. We use ResNet50 he2016deep image encoder for $F_{X} (\cdot)$

(trained from scratch or fine-tuned on ImageNet

deng2009imagenet pre-trained weights), BERT-uncased devlin2018bert text encoder for

F_{S} (\cdot)

(trained from scratch or BookCorpus zhu2015aligning /Wikipedia pre-trained weights), and a linear layer for

G (\cdot)

. After performing self-supervised visual-textual representation learning, we consider the downstream multi-label classification task across

91

categories. We evaluate learned visual representation (

Z_{X}

) using downstream linear evaluation protocol of oord2018representation ; henaff2019data ; tian2019contrastive ; hjelm2018learning ; bachman2019learning ; tschannen2019mutual . Specifically, a linear classifier is trained from the self-supervisedly learned (fixed) representation to the labels on the training set. Commonly used metrics for multi-label classification are reported on MS COCO validation set: Micro ROC-AUC, Hamming Loss, and Subset Accuracy. One may refer to Supplementary for more details on these metrics.

Figure 5:
Comparisons for different settings on self-supervised visual-textual representation training. We report metrics on MS COCO validation set with mean and standard deviation from
$5$ random trials. Micro ROC-AUC / Subset Accuracy are the higher the better and Hamming Loss is the lower the better.

$▹$ Results & Discussions. First, Figure 5 (a) suggests that the SSL strategy can work when the input and self-supervised signals lie in different modalities. For example, a random guess for the subset accuracy would be ${0.5}^{91} \approx 0$ , and the setting under Raw BERT + Raw ResNet achieves $0.0166$ . We also see that using pre-trained ResNet can further improve the self-supervisedly learned representation, while using pre-trained BERT does not give us obvious benefits. Next, Figure 5 (b) suggests that the self-supervisedly learned representations can be further improved by combining $L_{C L}$ and $L_{I P}$ : $L_{C L} + λ_{I P} L_{I P}$ . In Figure 5 (c)/(d), we have a similar observation as the self-supervised visual representation learning experiment: the hyper-parameter $λ_{I P}$ is sensitive to the performance.

4 Related Work

Our work aims at providing theoretical insights for the empirical success of self-supervised learning. The most related work is Unsupervised Contrastive Learning Theory arora2019theoretical that assumes two similar data (i.e., one stands for the input and the other stands for the corresponding self-supervised signal) have the same latent class, and a downstream classification task is comprised of a subset of the latent classes. Then, the work presented 1) provable guarantees for the downstream classification using contrastively learned representations; and 2) generalization bound such that the learned representations can reduce (labeled) sample complexity on downstream tasks. Our work differs in two ways: 1) we present a different assumption that only the shared information between the input and self-supervised signals contribute to the downstream tasks; and 2) we do not constrain the type of the downstream tasks to be classification, where they could be regression, clustering, etc.

Multi-view learning xu2013survey also closely relates to our work. Specifically, we can regard the input and self-supervised signals as two different views of data, and self-supervised learning aims at learning useful representations across views. Sridharan et. al. sridharan2008information pose the underlying assumption for multi-view learning: either view alone is sufficient for the downstream tasks (see Assumption 1 in sridharan2008information ). Their assumption is synonymous to our Assumption 1. Note that they focus on semi-supervised setting while we focus on unsupervised setting. Another recent work federici2020learning combines multi-view learning and information bottleneck tishby2000information method to balance the trade-off between extracting joint multi-view information and discarding non-joint multi-view information.

On empirical side, we explain why contrastive agrawal2015learning ; arandjelovic2017look ; jayaraman2015learning ; oord2018representation ; bachman2019learning ; chen2020simple ; tian2019contrastive ; hjelm2018learning ; he2019momentum ; kong2019mutual ; ozair2019wasserstein ; arora2019theoretical ; henaff2019data and predictive learning zhang2016colorful ; pathak2016context ; vondrick2016generating ; tulyakov2018mocogan ; srivastava2015unsupervised ; peters2018deep ; devlin2018bert ; dai2019transformer ; bai2018empirical approaches represent good self-supervised learning objectives, showing that these objectives can (unsupervisedly) extract task-relevant information.

5 Conclusion

In this paper, we studied self-supervised learning via an information-theoretical perspective. We designed a self-supervised learning framework to extract task-relevant information and discard task-irrelevant information. We also connected this framework with prior self-supervised learning methods, specifically for contrastive and predictive learning objectives. To support our theoretical analysis empirically, we designed controlled experiments on visual representation learning and visual-textual representation learning. We believe this work sheds light on the advantages of self-supervised learning and may help better understand when and why self-supervised learning is likely to work. In the future, we plan to investigate, compare, and combine different deployments of contrastive learning, forward predictive learning, and inverse predictive learning objectives. Another area of interest for future exploration is multi-modality self-supervised learning.

Acknowledgement

This work was supported in part by the DARPA grants FA875018C0150 HR00111990016, NSF IIS1763562, NSF Awards #1750439 #1722822, National Institutes of Health, and Apple. We would also like to acknowledge NVIDIA’s GPU support.

References

(1) Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE international conference on computer vision, pages 37–45, 2015.
(2) Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, pages 609–617, 2017.
(3) Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
(4) Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15509–15519, 2019.
(5) Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
(6) Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062, 2018.
(7) Cristian S Calude. Information and randomness: an algorithmic perspective. Springer Science & Business Media, 2013.
(8) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
(9) Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
(10) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
(11) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In
2009 IEEE conference on computer vision and pattern recognition
, pages 248–255. Ieee, 2009.
(12) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
(13) Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874, 2006.
(14) M Federici, A Dutta, P Forré, N Kushmann, and Z Akata. Learning robust representations via multi-view information bottleneck. International Conference on Learning Representation, 2020.
(15) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
(16) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
(17) Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019.
(18) R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.
(19) Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEEE International Conference on Computer Vision, pages 1413–1421, 2015.
(20) Lingpeng Kong, Cyprien de Masson d’Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. arXiv preprint arXiv:1910.08350, 2019.
(21) Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
(22) Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015.
(23) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
(24) Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
(25) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
(26) Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pages 15578–15588, 2019.
(27) Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
(28) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.
(29) Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint arXiv:1905.06922, 2019.
(30) Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.06222, 2019.
(31) Mohammad S Sorower. A literature survey on algorithms for multi-label learning.
(32) Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. 2008.
(33) Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In International conference on machine learning, pages 843–852, 2015.
(34) Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
(35) Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
(36) Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625, 2019.
(37) Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018.
(38) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
(39) Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems, pages 613–621, 2016.
(40) Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. arXiv preprint arXiv:1304.5634, 2013.
(41) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria.
Recent trends in deep learning based natural language processing.
ieee Computational intelligenCe magazine, 13(3):55–75, 2018.
(42) Richard Zhang, Phillip Isola, and Alexei A Efros.
Colorful image colorization.
In European conference on computer vision, pages 649–666. Springer, 2016.
(43) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015.

6 Proofs for Lemmas

Lemma 3 (Determinism, restating Lemma 1).

If $P (Z_{X} | X)$ is Dirac, then the following conditional independence holds: $T ⊥ ⊥ Z_{X} | X$ and $S ⊥ ⊥ Z_{X} | X$ , given by a Markov chain $S \leftrightarrow T \leftrightarrow X \to Z_{X}$ .

Proof.

When $Z_{X}$ is a deterministic function of $X$ , for any $A$ in the sigma-algebra induced by $Z_{X}$ we have $E [1_{[Z_{X} \in A]} | X, {T, S}] = E [1_{[Z_{X} \in A]} | X, S] = E [1_{[Z_{X} \in A]} | X]$ , which implies $T ⊥ ⊥ Z_{X} | X$ and $S ⊥ ⊥ Z_{X} | X$ . ∎

Bringing the redundancy assumption and determinism lemma together, we get:

Lemma 4 (Representation Redundancy).

The representation is redundant to the self-supervised signal for the task-relevant information, meaning $I (Z_{X}; T | S) = 0$ .

Proof.

By redundancy Assumption, $I (X; T | S) = 0$ . Also, $I (X; T | S) \geq I (F_{X} (X); T | S) = I (Z_{X}; T | S)$ . ∎

Lemma 5 (Supervision Decomposition, restating Lemma 2).

We consider the supervision decomposition from $T$ to $S$ :

I (Z_{X}; S) = I (Z_{X}; T) + I (Z_{X}; S | T) a n d H (Z_{X} | S) = H (Z_{X} | T) - I (Z_{X}; S | T) .

Also, $I (X; S) = I (X; T) + I (X; S | T) a n d H (X | S) = H (X | T) - I (X; S | T) .$

Proof.

Plug in $I (Z_{X}; T | S) = 0$ (see Lemma 4) into chain rules of mutual information: $I (Z_{X}; S) = I (Z_{X}; T) + I (Z_{X}; S | T) - I (Z_{X}; T | S)$ and $H (Z_{X} | S) = H (Z_{X} | T) - I (Z_{X}; S | T) + I (Z_{X}; T | S)$ . Likewise, plug in $I (X; T | S) = 0$ (see redundancy Assumption) into chain rules for $I (X; S)$ and $H (X | S)$ . ∎

7 Information Diagram Road Map

To ease the understanding of the paper, we provide an information-diagram version of our road map for our derivations. Note that information diagram provides easy-to-understand relationships between information measurements. We encourage the readers to refer to the main text for formal proofs and statements of the results.

At first, we introduce compressed supervised representation learning by minimizing $H (Z_{X} | T)$ and maximizing $I (Z_{X}; T)$ . This supervisedly learned representation contains only and no more than the task-relevant information, and hence is believed to be optimally compressed (for downstream tasks). Then, to connect with self-supervised learning, we perform a supervision transition from the downstream task to the self-supervised signal. Under some derivations, we show that minimizing $H (Z_{X} | S)$ is discarding task-irrelevant information and maximizing $I (Z_{X}; S)$ is extracting task-relevant information, even when these two objectives have no access to downstream tasks. The resulting optimally learned representation $Z_{X}^{*}$ contains only and no more than the shared information between $X$ / $S$ . Last, we demonstrate that $Z_{X}^{*}$ extracts all task-relevant information from $X / S$ and $I (X; S | T)$ is the information that cannot be discarded.

Our derivations are based on the following assumption and lemmas. The core assumption is that input and self-supervised signal are mutually redundant for downstream tasks. The assumption suggests the exclusive information in input and self-supervised signal is what we can discard. Next, using the fact that $Z_{X}$ is deterministic from $X$ , we characterize conditional independence by a Markov chain $S \leftrightarrow T \leftrightarrow X \to Z_{X}$ . This lemma simply states that post-processing (i.e., $X$ to $Z_{X}$ ) cannot introduce additional information. Last, based on the redundancy assumption and determinism lemma, we present supervision decomposition that is used for transiting supervision from the downstream task to the self-supervised signal.

After depicting our theories and their derivations, we connect our SSL framework and prior work zhang2016colorful ; devlin2018bert ; oord2018representation ; bachman2019learning ; chen2020simple ; tian2019contrastive ; hjelm2018learning ; he2019momentum , discussing practical implementation for different SSL objectives.

8 More on Visual Representation Learning Experiments

In the main text, we design controlled experiments on self-supervised visual representation learning to empirically support our theorem and examine different compositions of SSL objectives. In this section, we will discuss 1) the architecture design; 2) different deployments of contrastive/ forward predictive learning; and 3) different self-supervised signal construction strategy. We argue that these three additional set of experiments may be interesting future work.

8.1 Architecture Design

The input image has size $105 \times 105$ . For image augmentations, we adopt 1) rotation with degrees from $- 10^{\circ}$ to $+ 10^{\circ}$ ; 2) translation from $- 15$ pixels to $+ 15$ pixels; 3) scaling both width and height from $0.85$ to $1.0$ ; 4) scaling width from $0.85$ to $1.25$ while fixing the height; and 5) resizing the image to $28 \times 28$ . Then, a deep network takes a $28 \times 28$ image and outputs a $1024 -$ dim. feature vector. The deep network has the structure: $C o n v - B N - R e L U - C o n v - B N - R e L U - M a x P o o l - C o n v - B N - R e L U - M a x P o o l - C o n v$ $- B N - R e L U - M a x P o o l - F l a t t e n - L i n e a r - L 2 N o r m$ . $C o n v$ has 3x3 kernel size with $128$ output channels, $M a x P o o l$ has 2x2 kernel size, and $L i n e a r$ is a $1152$ to $1024$ weight matrix. $R (\cdot)$ is symmetric to $F_{X} (\cdot)$ , which has $L i n e a r - B N - R e L U - U n F l a t t e n - D e C o n v - B N - R e L U - D e C o n v - B N - R e L U - D e C o n v$ $- B N - R e L U - D e C o n v$ . $R (\cdot)$ has the exact same number of parameters as $F_{X} (\cdot)$ . Note that we use the same network designs in $I (\cdot, \cdot)$ and $H (\cdot | \cdot)$ estimations. To reproduce the results in our experimental section, please refer to our released code (https://github.com/yaohungt/Demystifying_Self_Supervised_Learning).

8.2 Different Deployments for Contrastive and Predictive Learning Objectives

In the main text, for practical deployments, we suggest Contrastive Predictive Coding (CPC) oord2018representation for $L_{C L}$ and assume Gaussian distribution for the variational distributions in $L_{F P}$ / $L_{I P}$ . The practical deployments can be abundant by using different mutual information approximations for $L_{C L}$ and having different distribution assumptions for $L_{F P}$ / $L_{I P}$ . In the following, we discuss a few examples.

Contrastive Learning. Other than CPC oord2018representation , another popular contrastive learning objective is JS bachman2019learning , which is the lower bound of Jensen-Shannon divergence between $P (Z_{S}, Z_{X})$ and $P (Z_{S}) P (Z_{X})$ (a variational bound of mutual information). Its objective can be written as

where we use $s o f t p l u s$ to denote $s o f t p l u s (x) = l o g (1 + e x p (x))$ .

Predictive Learning. Gaussian distribution may be the simplest distribution form that we can imagine, which leads to Mean Square Error (MSE) reconstruction loss. Here, we use forward predictive learning as an example, and we discuss the case when $S$ lies in discrete ${0, 1}$ sample space. Specifically, we let $Q_{ϕ} (S | Z_{X})$ be factorized multivariate Bernoulli:

m a x Z_{X} = F_{X} (X), R E_{P_{S, Z_{X}}} [p \sum i = 1 s_{i} \cdot l o g [R (z_{x})]_{i} + (1 - s_{i}) \cdot l o g [1 - R (z_{x})]_{i}] .

(5)

This objective leads to Binary Cross Entropy (BCE) reconstruction loss.

If we assume each reconstruction loss corresponds to a particular distribution form, then by ignoring which variatioinal distribution we choose, we are free to choose arbitrary reconstruction loss. For instance, by switching $s$ and $z$ in eq. (5), the objective can be regarded as Reverse Binary Cross Entropy Loss (RevBCE) reconstruction loss. In our experiments, we find RevBCE works the best among {MSE, BCE, and RevBCE}. Therefore, in the main text, we choose RevBCE as the example reconstruction loss as $L_{F P}$ .

More Experiments. We provide an additional set of experiments by having {CPC, JS} for $L_{C L}$ and {MSE, BCE, RevBCE} reconstruction loss for $L_{F P}$ in Figure 6. From the results, we find different formulation of objectives bring very different test generalization performance. We argue that, given a particular task, it is challenging but important to find the best deployments for contrastive and predictive learning objectives.

Figure 6: Comparisons for different objectives/compositions of SSL objectives on self-supervised visual representation training. We report mean and its standard error from $5$ random trials.

8.3 Different Self-supervised Signal Construction Strategy

Figure 7: Comparisons for different self-supervised signal construction strategies. The differences between the input and the self-supervised signals are {drawing styles, image augmentations} for our construction strategy and only {image augmentations} for SimCLR chen2020simple ’s strategy. We choose $L_{C L}$ as our objective, reporting mean and its standard error from $5$ random trials.

In the main text, we design a self-supervised signal construction strategy that the input ( $X$ ) and the self-supervised signal ( $S$ ) differ in {drawing styles, image augmentations}. This self-supervised signal construction strategy is different from the one that is commonly adopted in most self-supervised visual representation learning work tian2019contrastive ; bachman2019learning ; chen2020simple . Specifically, prior work consider the difference between input and the self-supervised signal only in image augmentations. We provide additional experiments in Fig. 7 to compare these two different self-supervised signal construction strategies.

We see that, comparing to the common self-supervised signal construction strategy tian2019contrastive ; bachman2019learning ; chen2020simple , the strategy introduced in our controlled experiments has much better generalization ability to test set. It is worth noting that, although our construction strategy has access to the label information (i.e., we sample the self-supervised signal image from the same character with the input image), our SSL objectives do not train with the labels. Nonetheless, since we implicitly utilize the label information in our self-supervised construction strategy, it will be unfair to directly compare our strategy and prior one. An interesting future research direction is examining different self-supervised signal construction strategy and even combine full/part of label information into self-supervised learning.

9 Metrics in Visual-Textual Representation Learning

Subset Accuracy $(A)$ multilbl , also know as the Exact Match Ratio (MR), ignores all partially correct (consider them incorrect) outputs and extend accuracy from the single label case to the multi-label setting.

$M R = \frac{1}{n} n \sum i = 1 1_{[Y_{i} = H_{i}]}$
Micro AUC ROC score aucroc computes the AUC (Area under the curve) of a receiver operating characteristic (ROC) curve.
Hamming Loss $(H L)$ multilbl is the fraction of wrong labels to the total number of labels.

$H L = \frac{1}{k n} n \sum i = 1 k \sum c = 1 1_{[Y_{i c} \neq H_{i c}]}$

Demystifying Self-Supervised Learning: An Information-Theoretical Framework

Authors

Self-supervised Learning: Generative or Contrastive

Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

i-Mix: A Strategy for Regularizing Contrastive Representation Learning

Automatic Shortcut Removal for Self-Supervised Representation Learning

Self-supervised visual feature learning with curriculum

Taxonomy of multimodal self-supervised representation learning

Self-Supervised Visual Representation Learning from Hierarchical Grouping

Code Repositories