What makes for good views for contrastive learning

1 Introduction

It is commonsense that how you look at an object does not change its identity. Nonetheless, Jorge Luis Borges imagined the alternative. In his short story on Funes the Memorious, the titular character becomes bothered that a “dog at three fourteen (seen from the side) should have the same name as the dog at three fifteen (seen from the front)” [6]. The curse of Funes is that he has a perfect memory, and every new way he looks at the world reveals a percept minutely distinct from anything he has seen before. He cannot collate the disparate experiences.

Most of us, fortunately, do not suffer from this curse. We build mental representations of identity that discard nuisances like time of day and viewing angle. The ability to build up view-invariant representations is central to a rich body of research on multiview learning. These methods seek representations of the world that are invariant to a family of viewing conditions. Currently, a popular paradigm is contrastive multiview learning, where two views of the same scene are brought together in representation space, and two views of different scenes are pushed apart.

This is a natural and powerful idea but it leaves open an important question: “which viewing conditions should we be invariant to?” It’s possible to go too far: if our task is to classify the time of day then we certainly should not use a representation that is invariant to time. Or, like Funes, we could go not far enough: representing each specific viewing angle independently would cripple our ability to track a dog as it moves about a scene.

We therefore seek representations with enough invariance to be robust to inconsequential variations but not so much as to discard information required by downstream tasks. In contrastive learning, the choice of “views” is what controls the information the representation captures, as the framework results in representations that focus on the shared information between views [50]. Views are commonly different sensory signals, like photos and sounds [3], or different image channels [65] or slices in time [68]

, but may also be different “augmented” versions of the same data tensor

[8]. If the shared information is small, then the learned representation can discard more information about the input and achieve a greater degree of invariance against nuisance variables. How can we find the right balance of views that share just the information we need, no more and no less?

We investigate this question in two ways. First, we demonstrate that the optimal choice of views depends critically on the downstream task. If you know the task, it is often possible to design effective views. Second, we empirically demonstrate that for many common ways of generating views, there is a sweet spot in terms of downstream performance where the mutual information (MI) between views is neither too high nor too low.

Our analysis suggests an “InfoMin principle”. A good set of views are those that share the minimal information necessary to perform well at the downstream task. This idea is related to the idea of minimal sufficient statistics [60] and the Information Bottleneck theory [67, 2], which have been previously articulated in the representation learning literature. This principle also complements the already popular “InfoMax principle” [42] , which states that a goal in representation learning is to capture as much information as possible about the stimulus. We argue that maximizing information is only useful in so far as that information is task-relevant. Beyond that point, learning representations that throw out information about nuisance variables is preferable as it can improve generalization and decrease sample complexity on downstream tasks [60].

Based on our findings, we also introduce a semi-supervised method to learn views that are effective for learning good representations when the downstream task is known. We additionally demonstrate that the InfoMin principle can be practically applied by simply seeking stronger data augmentation to further reduce mutual information toward the sweet spot. This effort results in state of the art accuracy on a standard benchmark.

Our contributions include:

Demonstrating that optimal views for contrastive representation learning are task-dependent.
Empirically finding a U-shaped relationship between an estimate of mutual information and representation quality in a variety of settings.
A new semi-supervised method to learn effective views for a given task.
Applying our understanding to achieve state of the art accuracy of $73.0 %$ on the ImageNet linear readout benchmark with a ResNet-50.

2 Related Work

Learning high-level representations of data that can be used to predict labels of interest is a well-studied problem in machine learning

[5]. In recent years, the most competitive methods for learning representations without labels have been self-supervised contrastive representation learning [50, 30, 73, 65, 61, 8]

. These methods use neural networks to learn a low-dimensional embedding of data by a “contrastive” loss which pushes apart dissimilar data pairs while pulling together similar pairs, an idea similar to exemplar learning

[19]. Models based on contrastive losses have significantly outperformed other approaches based on generative models, smoothness regularization, dense prediction [78, 37, 52, 65], and adversarial losses [18].

The core idea of contrastive representation learning is to learn a function (modeled by a deep network) that maps semantically nearby points (positive pairs) closer together in the embedding space, while pushing apart points that are dissimilar (negative pairs). One of the major design choices in contrastive learning is how to select the positive and negative pairs. For example, given a dataset of i.i.d. images, how can we synthesize positive and negative pairs?

The standard approach for generating positive pairs without additional annotations is to create multiple views of each datapoint. For example: splitting an image into luminance and chrominance [65], applying different random crops and data augmentations [73, 8, 4, 26, 75, 62], pasting an object into different backgrounds [79], using different timesteps within a video sequence [50, 80, 57, 25, 24], or using different patches within a single image [32, 50, 30]. Negative pairs can be generated by using views that come from randomly chosen images/patches/videos. In this work, we provide experimental evidence and analysis that can be used to guide the selection and learning of views.

Theoretically, we can think of the positive pairs as coming from a joint distribution over views:

p (v_{1}, v_{2})

, and the negative pairs as coming from a product of marginal distributions

p (v_{1}) p (v_{2})

. For the most popular contrastive loss, InfoNCE [50], the objective is then a lower bound on the mutual information between the two views:

I (v_{1}; v_{2})

. This connection between contrastive learning and mutual information maximization was first made in CPC [50] and is discussed further in [55]. However, recent work has called into question the interpretation of the success of the InfoNCE contrastive loss as information maximization [69], instead arguing that success is due to geometric requirements on the embedding space. Furthermore, theoretical and experimental work has highlighted that estimating mutual information in high dimensions is challenging and empirical work has shown that the InfoNCE and other bounds used in practice can be quite loose [43, 55, 51].

Leveraging labeled data in contrastive representation learning has been shown to guide representations towards task-relevant features that improve performance [76, 29, 4, 34, 72]. Here we leverage labeled data only to learn better views, and still perform contrastive representation learning using unlabeled data only. Future work could combine these approaches to leverage labels for both view learning and representation learning.

Recent work has begun to study and address the question we study here: what views lead to improved downstream accuracy? In [8], compositions of data augmentations were investigated for their effectiveness. Most similar to our work, a recent unpublished tech report [70] presents several desiderata for views in contrastive representation learning similar to our discussion of suffiiency and minimality, and presents new bounds on MI for alternative negative sampling schemes.

3 Preliminary: Contrastive Representation Learning

Let us consider the case when we are given two random variables

v_{1}

and

v_{2}

, and we wish to learn a parametric function to discriminate between samples from the empirical joint distribution

p (v_{1}) p (v_{2} | v_{1})

and samples from the product of marginals

p (v_{1}) p (v_{2})

. The resulting function is an estimator of the mutual information between

V_{1}

and

V_{2}

, and the InfoNCE loss [50] has been shown to maximize a lower bound on

I (v_{1}; v_{2})

. In practice, given an anchor point

v_{1, i}

, the InfoNCE loss is optimized to score the correct positive

v_{2, i} \sim p (v_{2} | v_{1, i})

higher compared to a set of

K

distractors

v_{2, j} \sim p (v_{2})

L_{N C E} = - E ⎡ ⎣ log \frac{e^{h (v_{1, i}, v_{2, i})}}{\sum_{j = 1}^{K} e^{h (v_{1, i}, v_{2, j})}} ⎤ ⎦

(1)

The score function $h (\cdot, \cdot)$ typically consists of two encoders ( $f_{v_{1}}$ and $f_{v_{2}}$ ) and a critic head $h$ . The two encoders $f_{v_{1}}$ and $f_{v_{2}}$ may or may not share parameters depending on whether $v_{1}$ and $v_{2}$ are from the same domain. Minimizing the above InfoNCE loss is equivalent to maximizing a lower bound (known as $I_{N C E} (v_{1}; v_{2})$ ) on the mutual information between $v_{1}$ and $v_{2}$ [50, 30], i.e.,

I (v_{1}; v_{2}) \geq log (K) - L_{N C E} = I_{N C E} (v_{1}; v_{2})

(2)

In practice, $v_{1}$ and $v_{2}$ are two views of the data. Despite engineering tricks, recent contrastive representation learning methods can be considered as different ways to construct $v_{1}$ and $v_{2}$ : (a) InsDis [73], AMDIM [4], MoCo [26] and SimCLR [8] apply data augmentation to the same image to obtain two crops as views; (b) CMC [65] employs natural image channels; (c) CPC [50] and CPCv2 [29] leverages spatial or temporal co-occurrence; (d) In the video domain, [64, 44, 39] use a video as $v_{1}$ and aligned texts as $v_{2}$ , while [47, 10, 53] leverages the correspondence between video and audio; (e) [66] even considers representations from teacher and student networks as $v_{1}$ and $v_{2}$ , and performs contrastive knowledge distillation.

4 What Are the Optimal Views for Contrastive Learning?

Figure 1: (a) Schematic of multiview contrastive representation learning, where a single image is split into two views, and passed through two separate encoders to learn an embedding where the views are close relative to views from other images. (b)-(c) As the mutual information between views is changed, information about the downstream task (green) and nuisance variables (red) can be selectively included or excluded, biasing the learned representation. (b) depicts a scenario where views are chosen to preserve downstream task information between views while throwing out nuisance information, while in (c) reducing MI always throws out information relevant for the task leading to decreasing performance as MI is reduced.

Given two views $v_{1}$ and $v_{2}$ of the data, the encoders $f_{1}$ and $f_{2}$ in the contrastive learning framework extract representations $z_{1} = f_{1} (v_{1})$ and $z_{2} = f_{2} (v_{2})$ , respectively.

(Sufficient Encoder) The encoder $f_{1}$ of $v_{1}$ is sufficient in the contrastive learning framework if and only if $I (v_{1}; v_{2}) = I (f_{1} (v_{1}); v_{2})$ .

Intuitively, the encoder $f_{1}$ is sufficient if the amount of information in $v_{1}$ regarding the contrastive objective is lossless during the encoding procedure. In other words, the representation $z_{1}$ has kept all the information about $v_{2}$ in $v_{1}$ , and therefore is as useful as $v_{1}$ . Symmetrically, we can also say $f_{2}$ is sufficient if $I (v_{1}; v_{2}) = I (v_{1}; f_{2} (v_{2}))$

(Minimal Sufficient Encoder) A sufficient encoder $f_{1}$ of $v_{1}$ is minimal if and only if $I (f_{1} (v_{1}); v_{1}) \leq I (f (v_{1}); v_{1}), \forall f$ that are sufficient.

Among those encoders which are sufficient, the minimal ones only extract relevant information of the contrastive task and will throw away other information. This is appealing in cases where the views are constructed in such a way that all the information we care about is shared between them.

The representations learned in the contrastive framework are typically used in a different downstream task. To characterize what representations are good for a downstream task, we define the optimality of representations. To make notation simple, we use $z$ to mean it can be either $z_{1}$ or $z_{2}$ . (Optimal Representation of a Task) For a task $T$ whose goal is to predict a semantic label $y$ from the input data $x$ , the optimal representation $z^{*}$ encoded from $x$ is the minimal sufficient statistic with respect to $y$ .

This says a model built on top of $z^{*}$ has all the information necessary to predict $y$ as accurately as if it were to access $x$ . Furthermore, $z^{*}$ maintains the smallest complexity, i.e., containing no other information besides that about $y$ , which makes it more generalizable [60]. We refer the reader to [60] for a more in depth discussion about optimal visual representations and minimal sufficient statistics.

4.1 InfoMin Principle: Views that Only Share Label Information

While views $v_{1}$ and $v_{2}$ can be arbitrarily constructed or selected from the data as we like, the effectiveness of views may vary in different tasks. For instance, views that share object position as mutual information should lead to better object localization performance in a downstream task compared to views that only share background nuisances. A general intuition is that good views should only share information that is relevant to a target downstream task. Then given these good input views, the minimal sufficient encoders will extract latent representation that only store information shared between them, which is task-relevant. The following InfoMin proposition articulates which views are optimal supposing that we know the specific downstream task $T$ in advance.

Proposition 4.1.

Suppose $f_{1}$ and $f_{2}$ in the contrastive learning framework are the minimal sufficient encoders. Given a downstream task $T$ associated with label $y$ , then the optimal views created from the complete data tensor $x$ would be $({v_{1}}^{*}, {v_{2}}^{*}) = {min}_{v_{1}, v_{2}} I (v_{1}; v_{2})$ , subject to . Given ${v_{1}}^{*}, {v_{2}}^{*}$ , the learned representations $z_{1}^{*}$ (or $z_{2}^{*}$ ) is optimal for $T$ (Definition 4), thanks to the minimality and sufficiency of $f_{1}$ and $f_{2}$ .

This InfoMin principle has two implications. First, we should reduce mutual information between views. By doing so, the minimal sufficient encoders will throw away more irrelevant information (or nuisance factors) to the downstream task. In practice, “shortcuts” are one type of task-irrelevant nuisance and should be removed from the views [45]. Furthermore, bits of information that are useful for a different downstream task $^T$ may turn into nuisances. Second, the constraints suggest that we should retain the predictability of $y$ from $x$ in both $v_{1}$ and $v_{2}$ , such that the representations capture the semantics of downstream task $T$ . This constraint dissolves the potential impression that we should make the contrastive task as hard as possible to obtain better representations. For a proof of this proposition, please refer to the Appendix.

An example of creating optimal views following this principle, in an image classification task, is to treat images from the same class as congruent pairs and images from different classes as incongruent pairs. In this way, congruent pairs of views only share label information. Recently in [34], such optimal views have been leveraged for supervised contrastive learning, outperforming supervised models trained with cross-entropy loss on ImageNet [14].

4.2 A Toy Example: Colorful Moving-MNIST

Figure 2: Illustration of the Colorful-Moving-MNIST dataset. In this example, the first view $v_{1}$ is a sequence of frames containing the moving digit, e.g., $v_{1} = x_{1 : k}$ . The matched second view $v_{2}^{+}$ share some factor with $x_{t}$ that $v_{1}$ can predict, while the unmatched view $v_{2}^{-}$ contains a random digit at a random position (with a random background).

Directly analyzing natural images can be challenging as it is hard to create interesting views whose factors of variation are controllable. Therefore, we use a toy dataset as a starting point to understand the behavior of contrastive representation learning with different views. Moving-MNIST [63] consists of videos where digits move inside a canvas with constant speed and bounce off of image boundaries. To simulate a more complex dataset with nuisance factors of variation we construct Colorful-Moving-MNIST by adding a background image to the Moving-MNIST videos. Concretely, given a video, a random image from STL-10 dataset [11] is selected, and then for each frame of the video we randomly crop a patch from this image as background. Thus the dataset consist of three factors of variation in each frame: the class of the digit, the position of the digit, and the class of background image.

Setup. While there are many ways to construct views, our goal is to analyze a set of easily reproducible experiments. To this end, we fix the first view $v_{1}$ as a sequence of past frames $x_{1 : k}$ , and construct different views $v_{2}$ . For simplicity, we consider $v_{2}$ as a single image. One example of constructing such views is shown in Fig. 2 where we use a future frame $x_{t}$ of the video as $v_{2}$ . During the contrastive learning phase, we employ a 4-layer ConvNet to encode images and use a single layer LSTM [31] on top of the ConvNet to aggregate features of continuous frames. After the contrastive pre-training phase, to read off what information has been encoded in the representation, we consider three different downstream tasks for an image: (1) predict the digit class; (2) localize the digit inside the canvas; (3) classify the background into one of the 10 classes of STL-10. In this transfer phase, we freeze the backbone network and learn a task-specific head. To facilitate comparison, we also provide a “supervised” baseline that is trained end-to-end using the same data.

I (v_{1}; v_{2})

digit cls.

error rate (%)

background cls.

error rate (%)

digit loc.

error pixels

Single

Factor

digit

16.8

88.6

13.6

bkgd

88.6

51.7

16.1

pos

57.9

87.6

3.95

Multiple

Factors

bkgd, digit, pos

88.8

56.3

16.2

bkgd, digit

88.2

53.9

16.3

bkgd, pos

88.8

53.8

15.9

digit, pos

14.5

88.9

13.7

Supervised

3.4

45.3

0.93

Table 1: We study how mutual information shared by views

I (v_{1}; v_{2})

would affect the representation quality. We evaluate the learned representation on three downstream tasks: digit classification, background (STL-10) classification, and digit localization.

Single Factor Shared. To begin with, we create views $v_{1}$ and $v_{2}$ that only share one of the three factors: digit, position, or background. For a video $x$ with $t$ frames, we set $v_{1} = x_{1 : k}$ (note $v_{1}$ can deterministically predict $x_{(k + 1) : t}$ ), and create another image as $v_{2}$ by setting one of the three factors the same as $x_{t}$ while randomly picking the other two. In such cases, $v_{1}$ can only deterministically predict one of the three factors in $v_{2}$ while never reduce the uncertainty of the other two factors. In other words, the mutual information $I (v_{1}; v_{2})$ is either digit, position, or background

. We separately train encoders using contrastive learning for each of the three scenarios, and the perform transfer learning in three downstream tasks, as shown in Table

1. These results clearly show that the performance of downstream tasks is significantly affected by the way we construct

v_{2}

, which determines

I (v_{1}; v_{2})

. Specifically, if the downstream task is relevant to one factor, we should let

I (v_{1}; v_{2})

include this factor rather than others. For example, when

v_{2}

only shares background image with

v_{1}

, contrastive learning can hardly learn representations that capture digit class and location. It is expected since the information of digit class and location is of no use to the contrastive pre-training objective, and thus will not be captured.

Multiple Factors Shared. A more interesting question is how representation quality is changed if $v_{1}$ and $v_{2}$ share multiple factors. We follow a similar procedure as above to control factors shared by $v_{1}$ and $v_{2}$ , and present the results in the second half of Table 1. We found that one factor can overwhelm another; for instance, whenever background is shared by the two views, the latent representation leaves out information for discriminating or localizing digits. This might be because the information bits of background easily dominate bits of the digit class and its position, and the network chooses the background as a “shortcut” to solve the contrastive pre-training task. When $v_{1}$ and $v_{2}$ share digit and position, we found two interesting observations: (1) digit dominates position as the digit localization task still performs poorly; (2) sharing position information benefits digit classification – $14.5 %$ error rate v.s. $16.8$ when only digit is shared. The former might not be a surprise as ConvNets are designed to be insensitive to position shift. For the latter, we conjecture that in practice the encoder is not sufficient so it will lose some bits of information about the digits, and knowing the position of the digits help it capture more bits about the digit class.

5 A Sweet Spot in Mutual Information: Reverse-U Shape

As suggested in Proposition 4.1, to obtain good performance on the downstream task, we should reduce the mutual information between views $I (v_{1}; v_{2})$ while retaining the task-relevant semantics. In other words, we should remove task-irrelevant information between views. In this section, we will first discuss a hypothesis for effect of $I (v_{1}; v_{2})$ on downstream transfer performance, and then empirically analyze three cases of reducing $I (v_{1}; v_{2})$ in practice.

5.1 Three Regimes of Information Captured

As both views are generated from the input $x$ , we can constrain the information between views $I (v_{1}; v_{2})$ by constraining how much information about $x$ is present in each view: $I (v_{1}; x)$ and $I (v_{2}; x)$ . Due to the data processing inequality, the information shared between views is bounded by the information contained about the input in each view: $I (v_{1}; v_{2}) \leq min (I (v_{1}; x), I (v_{2}; x))$ . As our representations $z_{1}, z_{2}$ are built from our views and learned by the contrastive objective with minimal sufficient encoders, the amount and type of shared between $v_{1}$ and $v_{2}$ (i.e., $I (v_{1}; v_{2})$ ) determines how well we perform on downstream tasks.

As in information bottleneck [67], we can trace out a tradeoff between how much information our views share about the input, and how well our learned representation performs at predicting a task-relevant variable $y$ . Depending on how our views are constructed, we may find that we are keeping too many irrelevant variables while discarding relevant variables (as in Figure. 1c), leading to suboptimal performance on the information plane. Alternatively, we can find the views that maximize $I (v_{1}; y)$ and $I (v_{2}; y)$ (how much information is contained about the task-relevant variable) while minimizing $I (v_{1}; v_{2})$ (how much information is shared about the input, including both task-relevant and irrelevant information). Even in the case of these optimal views, there are three regimes of performance we can consider that are depicted in Figure 3, and have been discussed previously in information bottleneck literature [67, 2, 21] :

missing information: When $I (v_{1}; v_{2}) < I (x; y)$ , there is information about the task-relevant variable that is discarded by the view, degrading performance.
sweet spot: When $I (v_{1}; y) = I (v_{2}; y) = I (v_{1}; v_{2}) = I (x; y)$ , the only information shared between $v_{1}$ and $v_{2}$ is task-relevant, and there is no irrelevant noise.
excess noise: As we increase the amount of information shared in the views beyond $I (x; y)$ , we begin to include additional information that is irrelevant for the downstream task. This can lead to worse generalization on the downstream task [2, 58].

We hypothesize that the best performing views will be close to the sweet spot: containing as much task-relevant information while discarding as much irrelevant information in the input as possible. Unlike in information bottleneck, for contrastive representation learning we often do not have access to a fully-labeled training set, and thus evaluating how much information about the task-relevant variable is contained in the representation at training time is challenging. Instead, the construction of views has typically been guided by domain knowledge that alters the input while preserving the task-relevant variable.

The above analysis suggests that transfer performance will be upper-bounded by a reverse-U shaped curve (Figure 3, right), with the sweet spot at the top of the curve. We next present a series of experiments that find such a curve in practical settings.

Figure 3: When we have views that maximize $I (v_{1}; y)$ and $I (v_{2}; y)$ (how much information is contained about the task-relevant variable) while minimizing $I (v_{1}; v_{2})$ (how much information is shared about the input, including both task-relevant and irrelevant information), there are three regimes: *missing information* which leads to degraded performance due to $I (v_{1}; v_{2}) < I (x; y)$ ; *excess noise* which worsens generalization due to additional noise; *sweet spot* where the only information shared between $v_{1}$ and $v_{2}$ is task-relevant and such information is complete.

5.2 Practical Cases of Reducing $I (v_{1}; v_{2})$

Often, the downstream task is unknown beforehand, and we may not know in which ways we should reduce $I (v_{1}; v_{2})$ to create views suitable for various downstream tasks. It is possible that both task-relevant signals and nuisance factors are reduced simultaneously, and therefore we do not have a guarantee on performance. But recently, [65] found a “reverse-U” shape phenomenon: reducing $I (v_{1}; v_{2})$ firstly leads to improved performance on the downstream task, then after a peak further decreasing $I (v_{1}; v_{2})$ causes performance degradation. This finding is in line with Hypothesis LABEL:as:learner: at the beginning of reducing $I (v_{1}; v_{2})$ , nuisance factors – the noise – are removed while most of the useful semantics – the signal – are preserved. Indeed this may be common in practice: data augmentation, of the proper magnitude, reduces $I (v_{1}; v_{2})$ but improves accuracy. Other than data augmentation, we show three examples where reduced $I (v_{1}; v_{2})$ leads to improved performance. We use $I_{N C E}$ as neural proxy for $I$ . Though it might not estimate $I$ accurately, it can still provide interesting analysis. We note here that within each plot with $I_{N C E}$ in this paper, we only vary the input views $v_{1}, v_{2}$ , but keep all other settings fixed, in order to make the plotted points directly comparable. For more implementation details, please refer to the appendix.

5.2.1 Reducing $I (v_{1}; v_{2})$ with Spatial Distance

Figure 4: We create views by using pairs of image patches at various offsets from each other. As $I_{N C E}$ decreases, the downstream task accuracy (STL-10 classification, CIFAR-10 classification) improves until $I_{N C E}$ hits a sweet spot. After that, reducing $I_{N C E}$ gradually lowers accuracy.

We create views by randomly cropping two patches of size 64x64 from the same image with fixed relative position. Namely, one patch starts at position $(x, y)$ while the other starts at $(x + d, y + d)$ , with $(x, y)$ being randomly generated. We increase $d$ from $64$ to $384$ , sampling from inside high resolution images (e.g., $\sim$ 2k pixels in each dimension) from the DIV2K dataset [1]. After contrastive training, we evaluate on STL-10 and CIFAR-10 by freezing the encoder and training a linear classification layer. The plots in Figure 4 shows the Mutual Information v.s. Accuracy. From natural image statistics we can expect that $I (v_{1}; v_{2})$ decreases when $d$ increases [59]. The plots demonstrate that this is also true, empirically, for the proxy $I_{N C E}$ , as $I_{N C E}$ decreases with increasin $d$ . Moreover, the results show replicate the “reverse-U” curve found in [65], and here we further show that this phenomenon is consistent across both STL-10 and CIFAR-10. We can identify the sweet spot at $d = 128$ .

5.2.2 Reducing $I (v_{1}; v_{2})$ with Different Color Spaces

Figure 5: We build views by splitting channels of different color spaces. As $I_{N C E}$ decreases, the accuracy on downstream tasks (STL-10 classification, NYU-v2 segmentation) improves. Whether the sweet spot has been touched is unclear.

The correlation between channels may vary significantly with different color spaces. Here we follow [65, 78] and split different color spaces into two views, such as ${Y, D b D r}, {R, G B}, {X, Y Z}$ , etc. We perform contrastive learning on STL-10, and measure the representation quality by training a linear classifier on the STL-10 dataset to perform image classification or a decoder head on NYU-V2 [48] images to perform semantic segmentation. As shown in Figure 5, the plots show downstream performance keeps increasing as $I_{N C E} (v_{1}; v_{2})$ decreases for both classification and segmentation. Here we do not observe any performance drop with these natural color spaces when reducing $I_{N C E} (v_{1}; v_{2})$ . In Section 6, we will show a learning method for new color space, which can further reduces $I_{N C E} (v_{1}; v_{2})$ to find the sweet spot, and go beyond it, where transfer performance begins again to drop.

5.2.3 Reducing $I (v_{1}; v_{2})$ with Frequency Separation

Figure 6: We create views by splitting images into low- and high-frequency pairs with a blur function parameterized by $σ$ . $I_{N C E}$ is maximized at $σ = 0.7$ . Starting from this point, either increasing or decreasing $σ$ will reduce $I_{N C E}$ but interestingly they form two different trajectories. When increasing $σ$ from 0.7, the accuracy firstly improves and then drops, forming a reverse-U shape corresponding to (b) in Figure 1. While decreasing $σ$ from 0.7, the accuracy keeps diminishing, corresponding to (c) in Figure 1.

Another example we consider is to separate images into low- and high-frequency images. To simplify, we extract $v_{1}$ and $v_{2}$ by Gaussian blur, i.e.,

	$v_{1}$	$= Blur (x, σ)$
	$v_{2}$	$= x - v_{1}$

where Blur is the Gaussian blur function and $σ$ is the parameter controlling the kernel. Extremely small or large $σ$ can make the high- or low-frequency image contain little information. In theory, the maximal $I (v_{1}; v_{2})$ is obtained with some intermediate $σ$ . As shown in Figure 6, we found $σ = 0.7$ leads to the maximal $I_{N C E}$ on the STL-10 dataset. Either blurring more or less will reduce $I_{N C E}$ , but interestingly blurring more leads to different trajectory in the plot than blurring less. When increasing $σ$ from 0.7, the accuracy firstly improves and then drops, forming a reverse-U shape with a sweet spot at $σ = 1.0$ . This situation corresponds to (b) in Figure 1. While decreasing $σ$ from 0.7, the accuracy keeps diminishing, corresponding to (c) in Figure 1. This reminds us of the two aspects in Proposition 4.1: mutual information is not the whole story; what information is shared between the two views also matters.

6 Synthesizing Effective Views

Figure 7: Schematic of contrastive representation learning with a learned view generator. An input image is split into two views using an invertible view generator. To learn the view generator, we optimize the losses in yellow: minimizing information between views while ensuring we can classify the object from each view. The encoders used to estimate mutual information are always trained to maximize the InfoNCE lower bound. After learning the view generator, we reset the weights of the encoders, and train with a fixed view generator without the additional supervised classification losses.

Can we learn novel views that reach the sweet spot by following the InfoMin principle? To explore this possibility, we design unsupervised and semi-supervised frameworks that learn novel views inspired by the InfoMin principle. Concretely, we extend the color space experiments in Section 5.2.2 by learning flow-based models that transfer natural color spaces into novel neural color spaces, from which we split channels to get views. After the views have been learned, we will do standard contrastive learning followed by linear classifier evaluation. In this section, we consider three methods: (1) random view generation which varies $I (v_{1}; v_{2})$ by constructing views with randomly initialized networks; (2) unsupervised view learning which reduces $I (v_{1}; v_{2})$ ; and (3) semi-supervised view learning which reduces $I (v_{1}; v_{2})$ while preserving task-relevant information. The idea of view learning is diagrammed in Figure 7.

6.1 Random Views

Figure 8: (a) We transform an image $X$ by a random initialized invertible flow model $g (\cdot)$ , and compare using ${X_{1}, X_{2 : 3}}$ as two views with ${g (X)_{1}, g (X)_{2 : 3}}$ . The transformed views have larger $I_{N C E}$ than the raw views, and result in a performance drop. (b) We only apply $g (\cdot)$ to the second view of the raw input $X_{2 : 3}$ . In theory, $I (X_{1}; X_{2 : 3})$ = $I (X_{1}; g (X_{2 : 3}))$ . Then we observe that $I_{N C E}$ of ${X_{1}, X_{2 : 3}}$ is almost identical to ${X_{1}, g (X_{2 : 3})}$ , and so are their downstream accuracies.

Flow-based generative models [16, 15, 36] are carefully-designed bijective functions between input images and latent space. Here we leverage this property to create random color spaces that are bijective and preserve total information. To do so, we restrict $g$ to be a pixel-wise (i.e., use 1x1 convolutions) flow. With a randomly initialized $g$ , an input image, $X$ , is transformed into $^X = g (X)$ , which has the same size as $X$ . We then split $^X$ over the channel dimension ${^X1,^X2:3}$ as two views for contrastive learning. By comparing ${^X1,^X2:3}$ with ${X_{1}, X_{2 : 3}}$ , we found that $I_{N C E}$ is typically increased after the transformation and the downstream accuracy drops, as shown in (a) of Figure 8.

Is it MI between views or the inductive bias of $g (\cdot)$ that plays the role for above spectrum plot? To check, we use $g (\cdot)$ to transform only half of the input, e.g., $X_{2 : 3}$ . Theoretically, the MI between views ${X_{1}, X_{2 : 3}}$ is identical to ${X_{1}, g (X_{2 : 3})}$ . In accord with theory, we found this is also true for the NCE estimation $I_{N C E}$ , as shown in (b) of Figure 8. Interestingly the classification accuracy of the downstream task using ${X_{1}, X_{2 : 3}}$ is almost the same as ${X_{1}, g (X_{2 : 3})}$ .

6.2 Unsupervised: Minimize $I (v_{1}; v_{2})$

The idea is to leverage an adversarial training strategy [23]. Given an input image $X$ , we transform it into $^X = g (X)$ . We train two encoders on top of ${^X}_{1}$ and ${^X}_{2 : 3}$ to maximize $I_{N C E} ({^X}_{1}; {^X}_{2 : 3})$ , similar to the discriminator in GAN [23]. Meanwhile, $g (\cdot)$ is adversarially trained to minimize $I_{N C E} ({^X}_{1}; {^X}_{2 : 3})$ . Formally, the objective is (shown as the bottom-left yellow box in Figure 7):

min g max f_{1}, f_{2} I_{N C E}^{f_{1}, f_{2}} (g (X)_{1}; g (X)_{2 : 3})

(3)

where $f_{1}, f_{2}$ correspond to $h (\cdot)$ in Eqn. 1. Alternatively, one may use other MI bounds [55]. In practice, we find $I_{N C E}$ works well and keep using it for simplicity. We note that the invertible and pixel-wise properties of $g (\cdot)$ prevent it from learning degenerate/trivial solutions.

Figure 9: View generator learned by (a) unsupervised or (b) semi-supervised objectives.

Implementation. This experiment is performed on STL-10. We try both volume preserving (VP) and non-volume preserving (NVP) flow models. $F$ and $G$ consist of $1$ x $1$ convolutional layers. For $f_{1}$ and $f_{2}$ , we use an AlexNet-style network. We experiment with two input color spaces: RGB and YDbDr. The former is the most widely used one, while the latter is the best for contrastive learning, as shown in Figure 5. Results. We plot the $I_{N C E}$ between the learned views and the corresponding linear evaluation performance. As shown in Figure 9(a), a reverse U-shape between $I_{N C E}$ and downstream accuracy is presented. Interestingly, YDbDr color space is already around the sweet spot, and further reducing $I_{N C E}$ results in a performance drop. This happens to be in line with our human prior that the “luminance-chrominance” decomposition is a good way to decorrelate colors but still maintains good interpretability (in the sense that we can still read out high level semantics to perform tasks). We also note that the Lab color space, which is another luminance-chrominance decomposition (and performs similarly well to YDbDr; Figure 5), was designed to mimic the way humans perceive color [33]. Our analysis therefore suggests yet another rational explanation for why humans perceive color the way we do – human perception of color may be near optimal for self-supervised representation learning.

Occasionally, one color space learned from RGB happens to touch the sweet spot, but in general the $I_{N C E}$ between views is overly decreased. The reverse-U shape trend holds for both NVP and VP models. In addition, we found this GAN-style training is unstable, as different runs with the same hyper-parameter can vary a lot. We conjecture that while reducing MI between views in such an unsupervised manner, the view generator has no knowledge about task-relevant semantics and thus construct views that do not share sufficient information about the label $y$ , i.e., the constraint in Proposition 4.1 is not satisfied. To overcome this, we further develop an semi-supervised view learning method.

6.3 Semi-supervised View Learning: Find Views that Share the Label Information

We assume a handful of labels for the downstream task are available. Therefore we can teach the view generator $g (\cdot)$ to retain $I (g (X)_{1}, y)$ and $I (g (X)_{2 : 3}, y)$ as much as possible. Put into practice, we introduce two classifiers on each of the learned views to perform classification during the view learning process. Formally, we optimize (shown as the three yellow boxes in Figure 7):

min g, c_{1}, c_{2} max f_{1}, f_{2} I_{N C E}^{f_{1}, f_{2}} (g (X)_{1}; g (X)_{2 : 3})      unsupervised: reduce I (v_{1}; v_{2}) + L_{c e} (c_{1} (g (X)_{1}), y) + L_{c e} (c_{2} (g (X)_{2 : 3}), y)      supervised: keep I (v_{1}; y) and I (v_{2}; y)

(4)

where $c_{1}, c_{2}$ represent the classifiers. The $I_{N C E}$ term applies to all data while the latter two are only for labeled data. After this process is done, we use $g (\cdot)$ to generate views for contrastive representation learning.

Table 3: Switching to larger backbones with views learned by the semi-supervised method.

Method	RGB	YDbDr
unsupervised	82.4 $\pm$ 3.2	84.3 $\pm$ 0.5
supervised	79.9 $\pm$ 1.5	78.5 $\pm$ 2.3
semi-supervised	86.0 $\pm$ 0.6	87.0 $\pm$ 0.3
raw input	81.5 $\pm$ 0.2	86.6 $\pm$ 0.2

Results. The plots are shown in Figure 9(b). Now the learned views are clustered around the sweet spot, no matter what the input color space was and whether the generator is VP or NVP, which highlights the importance of keeping information about $y$ . Meanwhile, to see the importance of the unsupervised term, which reduces $I_{N C E}$ , we train another view generator by just the supervised loss. We further compare “supervised”, “unsupervised” and “semi-supervised” (the supervised + unsupervised losses) generators in Table 3, where we also includes contrastive learning over the original color space (“raw input”) as a baseline. The semi-supervised view generator significantly outperforms the supervised one, verifying the importance of reducing $I (v_{1}; v_{2})$ . We compare the “semi-supervised” views of $g (X)$ with the original $X$ ( $X$ is RGB or YDbDr) on larger backbone networks, as shown in Table 3. We see that the learned views consistently outperform its raw input, e.g., $g (R G B)$ surpasses $R G B$ by a large margin and reaches $94 %$ classification accuracy.

7 Data Augmentation as InfoMin

7.1 A Unified View of Recent State-of-the-Art Methods

Recently, there are several contrastive learning methods dominating the ImageNet self-suspervised learning benchmark, e.g., InstDis, CPC, CMC, MoCo, PIRL, SimCLR, etc. Despite various different engineering tricks in each paper, each has created $v_{1}$ and $v_{1}$ in a way that implicitly follows the InfoMin principle. We examine a non-comprehensive selection of recent methods below:

InstDis [73] and MoCo [26].

Despite the difference of memory bank and moment encoder, these two methods create views by ancestral sampling: (1) sample a image

X

from the empirical distribution

p (x)

; (2) sample two independent transformations

t_{1}, t_{2}

from a distribution of data augmentation functions

T

; (3) let

v_{1} = t_{1} (X)

and

v_{2} = t_{2} (X)

CMC [65]. On top of the two views $v_{1}$ and $v_{2}$ in InsDis, CMC further split images across color channels. This leads to a new set of views $v_{1}^{c m c}$ and $v_{2}^{c m c}$ , where $v_{1}^{c m c}$ is the first color channel of $v_{1}$ , and $v_{2}^{c m c}$ is the last two channels of $v_{2}$ . By this design, $I (v_{1}^{c m c}; v_{2}^{c m c}) \leq I (v_{1}; v_{2})$ is theoretically guaranteed, and we observe that CMC performs better than InstDis.

PIRL [46]. Comparing PIRL with InstDis is a bit tricky, but we can also explain it from the InfoMin perspective. Similarly, given two views obtained in InstDis, PIRL keeps $v_{1}^{P I R L} = v_{1}$ but transforms the other view $v_{2}$ with random JigSaw shuffling $h$ to get $v_{2}^{P I R L} = h (v_{2})$ . The mutual information between the two views is also reduced as $h (\cdot)$ introduces randomness.

SimCLR [8]. Despite other engineering techniques and tricks, the way SimCLR creates views is most similar to InstDist and MoCo, but it uses a stronger class of augmentations $T^{'}$ , which leads to less mutual information between the two views.

CPC [50]. Different from the above methods that create views at the image level, CPC extracts views $v_{1}^{c p c}$ , $v_{2}^{c p c}$ from local patches with strong data augmentation (e.g., RandAugment [13]) which reduces $I (v_{1}^{c p c}; v_{2}^{c p c})$ . In addition, cropping $v_{1}^{c p c}$ and $v_{2}^{c p c}$ from disjoint patches also reduces $I (v_{1}^{c p c}; v_{2}^{c p c})$ , which relates to the discussion in Section 5.2.1.

While the above methods are reducing mutual information between views, they keep information about object identity in both views. The hope is that object identity can bake in most of the high-level semantic information which various downstream tasks care about.

Figure 10: (a) data augmentation as InfoMin on ImageNet with linear projection head; (b) illustration of step-by-step data augmentation used in InfoMin.

7.2 Analysis of Data Augmentation as it relates to MI and Transfer Performance

We gradually strengthen the family of data augmentation functions $T$ , and plot the trend between accuracy in downstream linear evaluation benchmarks and $I_{N C E}$ . The overall results are shown in Figure 10(a), where the plot is generated by only varying data augmentation while keeping all other settings fixed. We consider Color Jittering with various strengths, Gaussian Blur, RandAugment [13], and their combinations, as illustrated in Figure 10(b). The results suggest that as we reduce $I_{N C E} (v_{1}; v_{1})$ , via stronger $T$ (in theory, $I (v_{1}; v_{1})$ also decreases), the downstream accuracy keeps improving.

We also investigate how sliding the strength parameter of individual augmentation functions leads to a practical reverse-U curves, as shown in Figures 11 and 12.

Cropping.

In PyTorch, the

RandomResizedCrop(scale=(c, 1.0)) data augmentation function sets a low-area cropping bound c. Smaller c means more aggressive data augmentation. We vary c for both a linear critic head [73] (with temperature 0.07) and nonlinear critic head [8] (with temperature 0.15), as shown in Figure 11. In both cases, decreasing c forms a reverse-U shape between

I_{N C E}

and linear classification accuracy, with a sweet spot at

c = 0.2

. This is different from the widely used

0.08

in the supervised learning setting. Using

0.08

can lead to more than

1 %

drop in accuracy compared to the optimal

0.2

when a nonlinear projection head is applied.

Color Jittering. As shown in Figure 10(b), we adopt a parameter $x$ to control the strengths of color jittering function. As shown in Figure 12, increasing $x$ from $0.125$ to $2.5$ also traces a reverse-U shape, no matter whether a linear or nonlinear projection head is used. The sweet spot lies around $x = 1.0$ , which is the same value as used in SimCLR [8]. Practically, we see the accuracy is more sensitive around the sweet spot for the nonlinear projection head, which also happens for cropping. This implies that it is important to find the sweet spot for future design of augmentation functions.

Details. These plots are based on the MoCo [26] framework. We use $65536$

negatives and pre-train for 100 epochs on 8 GPUs with a batch size of 256. The learning rate starts as

0.03

and decays following a cosine annealing schedule. For the downstream task of linear evaluation, we train the linear classifier for 60 epochs with an initial learning rate of 30, following [65].

7.3 Results on ImageNet Benchmark

Method	Architecture	Param.	Head	Epochs	Top-1	Top-5
InstDis [73]	ResNet-50	24	Linear	200	56.5	-
Local Agg. [81]	ResNet-50	24	Linear	200	58.8	-
CMC [65]	ResNet-50*	12	Linear	240	60.0	82.3
MoCo [26]	ResNet-50	24	Linear	200	60.6	-
PIRL [46]	ResNet-50	24	Linear	800	63.6	-
CPC v2 [29]	ResNet-50	24	-	-	63.8	85.3
SimCLR [8]	ResNet-50	24	MLP	1000	69.3	89.0
InfoMin Aug. (Ours)	ResNet-50	24	MLP	100	67.4	87.9
InfoMin Aug. (Ours)	ResNet-50	24	MLP	200	70.1	89.4
InfoMin Aug. (Ours)	ResNet-50	24	MLP	800	73.0	91.1

Table 4: Single-crop ImageNet accuracies (%) of linear classifiers [77] trained on representations learned with different contrastive methods using ResNet-50 [28]. InfoMin Aug. refers to data augmentation using RandomResizedCrop, Color Jittering, Gaussian Blur, RandAugment, Color Dropping, and a JigSaw branch as in PIRL [46]. * indicates splitting the network into two halves.

On top of the “RA-CJ-Blur” augmentations shown in Figure 10, we further reduce the mutual information (or enhance the invariance) of views by using PIRL [46], i.e., adding JigSaw [49]. This improves the accuracy of the linear classifier from $63.6 %$ to $65.9 %$ . Replacing the widely-used linear projection head [73, 65, 26] with a 2-layer MLP [8] increases the accuracy to $67.3 %$ . When using this nonlinear projection head, we found a larger temperature is beneficial for downstream linear readout (as also reported in [9]). All these numbers are obtained with 100 epochs of pre-training. For simplicity, we call such unsupervised pre-training as InfoMin pre-training (i.e., pre-training with our InfoMin inspired augmentation). As shown in Table 4, our InfoMin model trained with 200 epochs achieves $70.1 %$ , outperforming SimCLR with 1000 epochs. Finally, a new state-of-the-art, $73.0 %$ is obtained by training for 800 epochs. Compared to SimCLR requiring 128 TPUs for large batch training, our model can be trained with as less as 4 GPUs on a single machine.

For future improvement, there is still room for manually designing better data augmentation. As shown in Figure 10(a), using “RA-CJ-Blur” has not touched the sweet spot yet. Another way to is to learn to synthesize better views (augmentations) by following (and expanding) the idea of semi-supervised view learning method presented in Section 6.3.

7.4 Transferring Representations

One goal of unsupervised pre-training is to learn transferable representations that are beneficial for downstream tasks. The rapid progress of many vision tasks in past years can be ascribed to the paradigm of fine-tuning models that are initialized from supervised pre-training on ImageNet. When transferring to PASCAL VOC [20] and COCO [41], we found our InfoMin pre-training consistently outperforms supervised pre-training as well as other unsupervised pre-training methods.

Feature normalization has been shown to be important during fine-tuning [26]. Therefore, we fine-tune the backbone with Synchronized BN (SyncBN [54]) and add SyncBN to newly initialized layers (e.g., FPN [40]). Table 5 reports the bounding box AP and mask AP on val2017 on COCO, using the Mask R-CNN [27] R50-FPN pipeline. All results are reported on Detectron2 [71]. We notice that, among unsupervised approaches, only ours consistently outperforms supervised pre-training.

pre-train	AP $_{~{}}^{bb}$	AP $_{50}^{bb}$	AP $_{75}^{bb}$	AP $_{~{}}^{mk}$	AP $_{50}^{mk}$	AP $_{75}^{mk}$
random init	32.8	50.9	35.3	29.9	47.9	32.0
supervised	39.7	59.5	43.3	35.9	56.6	38.6
InstDis [73]	38.8( $↓$ 0.9)	58.4( $↓$ 1.1)	42.5( $↓$ 0.8)	35.2( $↓$ 0.7)	55.8( $↓$ 0.8)	37.8( $↓$ 0.8)
PIRL [46]	38.6( $↓$ 1.1)	58.2( $↓$ 1.3)	42.1( $↓$ 1.2)	35.1( $↓$ 0.8)	55.5( $↓$ 1.1)	37.7( $↓$ 0.9)
MoCo [26]	39.4( $↓$ 0.3)	59.1( $↓$ 0.4)	42.9( $↓$ 0.4)	35.6( $↓$ 0.3)	56.2( $↓$ 0.4)	38.0( $↓$ 0.6)
InfoMin Aug.	40.6( $↑$ 0.9)	60.6( $↑$ 1.1)	44.6( $↑$ 1.3)	36.7( $↑$ 0.8)	57.7( $↑$ 1.1)	39.4( $↑$ 0.8)

(a) Mask R-CNN, R50-FPN, 1

x

schedule

pre-train	AP $_{~{}}^{bb}$	AP $_{50}^{bb}$	AP $_{75}^{bb}$	AP $_{~{}}^{mk}$	AP $_{50}^{mk}$	AP $_{75}^{mk}$
random init	38.4	57.5	42.0	34.7	54.8	37.2
supervised	41.6	61.7	45.3	37.6	58.7	40.4
InstDis [73]	41.3( $↓$ 0.3)	61.0( $↓$ 0.7)	45.3( $↓$ 0.0)	37.3( $↓$ 0.3)	58.3( $↓$ 0.4)	39.9( $↓$ 0.5)
PIRL [46]	41.2( $↓$ 0.4)	61.2( $↓$ 0.5)	45.2( $↓$ 0.1)	37.4( $↓$ 0.2)	58.5( $↓$ 0.2)	40.3( $↓$ 0.1)
MoCo [26]	41.7( $↑$ 0.1)	61.4( $↓$ 0.3)	45.7( $↑$ 0.4)	37.5( $↓$ 0.1)	58.6( $↓$ 0.1)	40.5( $↑$ 0.1)
InfoMin Aug.	42.5( $↑$ 0.9)	62.7( $↑$ 1.0)	46.8( $↑$ 1.5)	38.4( $↑$ 0.8)	59.7( $↑$ 1.0)	41.4( $↑$ 1.0)

(b) Mask R-CNN, R50-FPN, 2

x

schedule

Table 5: Results of object detection and instance segmentation fine-tuned on COCO. We adopt Mask R-CNN R50-FPN, and report the bounding box AP and mask AP on val2017. In the brackets are the gaps to the ImageNet supervised pre-training counterpart. For fair comparison, InstDis [73], PIRL [46], MoCo [26], and InfoMin are all pre-trained for 200 epochs.

We have tried different popular detection frameworks with various backbones, extended the fine-tuning schedule (e.g., 6 $x$ schedule), and compared InfoMin ResNeXt-152 [74] trained on ImageNet-1k with supervised ResNeXt-152 trained on ImageNet-5k (6 times larger than ImageNet-1k). In all cases, InfoMin consistently outperforms supervised pre-training. For further details on these results, as well as experiments on PASCAL VOC, please refer to the appendix.

8 Conclusion

We have proposed an InfoMin principle and a view synthesis framework for constructing effective views for contrastive representation learning. Viewing data augmentation as information minimization, we achieved a new state-of-the-art result on the ImageNet linear readout benchmark with a ResNet-50.

Acknowledgments.

This work was done when Yonglong Tian was a student researcher at Google. We thank Kevin Murphy for fruitful and insightful discussion; Lucas Beyer for feedback on the manuscript; and Google Cloud team for supporting computation resources. Yonglong is grateful to Zhoutong Zhang for encouragement and feedback on experimental design.

References

[1] Eirikur Agustsson and Radu Timofte.
Ntire 2017 challenge on single image super-resolution: Dataset and study.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
, pages 126–135, 2017.
[2] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
[3] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), pages 435–451, 2018.
[4] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910, 2019.
[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
[6] Jorge Luis Borges. Funes, the memorious. na, 1962.
[7] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
[9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[10] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang. Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. In ICASSP, 2019.
[11] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In
Proceedings of the fourteenth international conference on artificial intelligence and statistics
, pages 215–223, 2011.
[12] Thomas M Cover and Joy A Thomas. Entropy, relative entropy and mutual information. Elements of information theory, 2:1–55, 1991.
[13] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719, 2019.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009.
[15] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
[16] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.
[17] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
[18] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pages 10541–10551, 2019.
[19] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.
Discriminative unsupervised feature learning with convolutional neural networks.
In NIPS, 2014.
[20] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 2010.
[21] Ian Fischer. The conditional entropy bottleneck. arXiv preprint arXiv:2002.05379, 2020.
[22] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
[24] Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.07990, 2020.
[25] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV Workshop, 2019.
[26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
[27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2017.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[29] Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019.
[30] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
[31] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
[32] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson. Learning visual groups from co-occurrences in space and time. International Conference on Learning Representations (ICLR), Workshop track, 2016.
[33] Anil K Jain. Fundamentals of digital image processing. Englewood Cliffs, NJ: Prentice Hall,, 1989.
[34] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11362, 2020.
[35] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[36] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, 2018.
[37] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[38] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2019.
[39] Tianhao Li and Limin Wang. Learning spatiotemporal features via video and text pair discrimination. arXiv preprint arXiv:2001.05691, 2020.
[40] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
[41] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014.
[42] Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):105–117, 1988.
[43] David McAllester and Karl Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04251, 2018.
[44] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. arXiv preprint arXiv:1912.06430, 2019.
[45] Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. arXiv preprint arXiv:2002.08822, 2020.
[46] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv preprint arXiv:1912.01991, 2019.
[47] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004.12943, 2020.
[48] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[49] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
[50] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
[51] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191–1253, 2003.
[52] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
[53] Mandela Patrick, Yuki M Asano, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003.04298, 2020.
[54] Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[55] Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint arXiv:1905.06922, 2019.
[56] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 2015.
[57] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018.
[58] Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411(29-30):2696–2711, 2010.
[59] Eero P Simoncelli. 4.7 statistical modeling of photographic images. Handbook of Video and Image Processing, 2005.
[60] Stefano Soatto and Alessandro Chiuso. Visual representations: Defining properties and deep approximations. In ICLR, 2016.
[61] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
[62] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136, 2020.
[63] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In International conference on machine learning, 2015.
[64] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743, 2019.
[65] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
[66] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[67] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[68] Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. arXiv preprint arXiv:1912.02783, 2019.
[69] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625, 2019.
[70] Mike Wu, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. On the importance of views in unsupervised representation learning. preprint, 2020.
[71] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2, 2019.
[72] Zhirong Wu, Alexei A Efros, and Stella X Yu. Improving generalization via scalable neighborhood component analysis. In ECCV, 2018.
[73] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
[74] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
[75] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[76] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer.
S4l: Self-supervised semi-supervised learning.
In Proceedings of the IEEE international conference on computer vision, pages 1476–1485, 2019.
[77] Richard Zhang, Phillip Isola, and Alexei A Efros.
Colorful image colorization.
In European conference on computer vision, pages 649–666. Springer, 2016.
[78] Richard Zhang, Phillip Isola, and Alexei A Efros.
Split-brain autoencoders: Unsupervised learning by cross-channel prediction.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058–1067, 2017.
[79] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. Distilling localization for self-supervised representation learning. arXiv preprint arXiv:2004.06638, 2020.
[80] Chengxu Zhuang, Alex Andonian, and Daniel Yamins. Unsupervised learning from video with deep neural embeddings. arXiv preprint arXiv:1905.11954, 2019.
[81] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. arXiv preprint arXiv:1903.12355, 2019.

Appendix A Proof of Proposition 1

In this section, we provide proof for the statement regarding optimal views in proposition 1 of the main text. As a warmup, we firstly recap some properties of mutual information.

a.1 Properties of MI [12]:

(1) Nonnegativity:

I (x; y) \geq 0; I (x; y | z) \geq 0

(2) Chain Rule:

I (x; y, z) = I (x; y) + I (x; z | y)

(2) Multivariate Mutual Information:

I (x_{1}; x_{2}; . . .; x_{n + 1}) = I (x_{1}; . . .; x_{n}) - I (x_{1}; . . .; x_{n} | x_{n + 1})

a.2 Proof

Proposition A.1.

According to Proposition 1, the optimal views $v_{1}^{*}, v_{2}^{*}$ for task $T$ with label $y$ , are views such that $I (v_{1}^{*}; v_{2}^{*}) = I (v_{1}^{*}; y) = I (v_{2}^{*}; y) = I (x; y)$

Proof.

Since $I (v_{1}; y) = I (v_{2}; y) = I (x; y)$ , and $v_{1}$ , $v_{2}$ are functions of $x$ .

	$I (y; x)$	$= I (y; v_{1}, v_{2})$
		$= I (y; v_{1}) + I (y; v_{2} \| v_{1})$
		$= I (y; x) + I (y; v_{2} \| v_{1})$

Therefore $I (y; v_{2} | v_{1}) = 0$ , due to the nonnegativity. Then we have:

	$I (v_{1}; v_{2})$	$= I (v_{1}; v_{2}) + I (y; v_{2} \| v_{1})$
		$= I (v_{2}; v_{1}, y)$
		$= I (v_{2}; y) + I (v_{2}; v_{1} \| y)$
		$\geq I (v_{2}; y) = I (x; y)$

Therefore the optimal views $v_{1}^{*}, v_{2}^{*}$ that minimizes $I (v_{1}; v_{2})$ subject to the constraint yields $I (v_{1}^{*}; v_{2}^{*}) = I (x; y)$ . Also note that optimal views $v_{1}^{*}, v_{2}^{*}$ are conditionally independent given $y$ , as now $I (v_{2}^{*}; v_{1}^{*} | y) = 0$ . ∎

Proposition A.2.

Given optimal views $v_{1}^{*}, v_{2}^{*}$ and minimal sufficient encoders $f_{1}$ , $f_{2}$ , then the learned representations $z_{1}$ $($ or $z_{2})$ are sufficient statistic of $v_{1}$ $($ or $v_{2})$ for $y$ , i.e., $I (z_{1}; y) = I (v_{1}; y)$ or $I (z_{2}; y) = I (v_{2}; y)$ .

Proof.

Let’s prove for $z_{1}$ . Since $z_{1}$ is a function of $v_{1}$ , we have:

	$I (y; v_{1})$	$= I (y; v_{1}, z_{1})$
		$= I (y; z_{1}) + I (y; v_{1} \| z_{1})$

To prove $I (y; v_{1}) = I (y; z_{1})$ , we need to prove $I (y; v_{1} | z_{1}) = 0$ .

	$I (y; v_{1} \| z_{1})$	$= I (y; v_{1}) - I (y; v_{1}; z_{1})$
		$= I (y; v_{1}; v_{2}) + I (y; v_{1} \| v_{2}) - I (y; v_{1}; z_{1})$
		$= I (y; v_{1}; v_{2}) + I (y; v_{1} \| v_{2}) - [I (y; v_{1}; z_{1}; v_{2}) + I (y; v_{1}; z_{1} \| v_{2})]$
		$= I (y; v_{1} \| v_{2}) + [I (y; v_{1}; v_{2}) - I (y; v_{1}; z_{1}; v_{2})] - I (y; v_{1}; z_{1} \| v_{2})$
		$= I (y; v_{1} \| v_{2}) + I (y; v_{1}; v_{2} \| z_{1}) - I (y; v_{1}; z_{1} \| v_{2})$
		$= I (y; v_{1} \| v_{2}) + I (y; v_{1}; v_{2} \| z_{1}) + I (y; z_{1} \| v_{1}, v_{2})      0 - I (y; z_{1} \| v_{2})$

		$= I (y; v_{1} \| v_{2}) + I (v_{1}; v_{2} \| z_{1}) - I (v_{1}; v_{2} \| y, z_{1})      0$
		$= I (y; v_{1} \| v_{2}) + I (v_{1}; v_{2} \| z_{1})$

In the above derivation $I (y; z_{1} | v_{1}, v_{2}) = 0$ because $z_{1}$ is a function of $v_{1}$ ; $I (v_{1}; v_{2} | y, z_{1}) = 0$ because optimal views $v_{1}, v_{2}$ are conditional independent given $y$ , see Proposition A.1. Now, we can easily prove $I (y; v_{1} | v_{2}) = 0$ following a similar procedure in Proposition A.1. If we can further prove $I (v_{1}; v_{2} | z_{1}) = 0$ , then we get $I (y; v_{1} | z_{1}) \leq 0$ . By nonnegativity, we will have $I (y; v_{1} | z_{1}) = 0$ .

To see $I (v_{1}; v_{2} | z_{1}) = 0$ , recall that our encoders are sufficient. According to Definition 1, we have $I (v_{1}; v_{2}) = I (v_{2}; z_{1})$ :

	$I (v_{1}; v_{2} \| z_{1})$

		$= 0$

∎

Proposition A.3.

The representations $z_{1}$ and $z_{2}$ are also minimal for $y$ .

Proof.

For all sufficient encoders, we have proved $z_{1}$ are sufficient statistic of $v_{1}$ for predicting $y$ . Namely $I (v_{1}; y | z_{1}) = 0$ . Now:

	$I (z_{1}; v_{1})$	$= I (z_{1}; v_{1} \| y) + I (z_{1}; v_{1}; y)$
		$= I (z_{1}; v_{1} \| y) + I (v_{1}; y) - I (v_{1}; y \| z_{1})      0$
		$= I (z_{1}; v_{1} \| y) + I (v_{1}; y)$
		$\geq I (v_{1}; y)$

The minimal sufficient encoder will minimize $I (z_{1}; v_{1})$ to $I (v_{1}; y)$ . This is achievable and leads to $I (z_{1}; v_{1} | y) = 0$ . Therefore, $z_{1}$ is a minimal sufficient statistic for predicting $y$ , thus optimal. Similarly, $z_{2}$ is also optimal. ∎

Appendix B Implementation Details

b.1 Colorful Moving MNIST

Dataset. Following the original Moving MNIST dataset [63], we use a canvas of size 64 $\times$ 64, which contains a digit of size 28 $\times$ 28. The back ground image is a random crop from original STL-10 images (96 $\times$ 96). The starting position of the digit is uniformly sampled inside the canvas. The direction of the moving velocity is uniformly sampled in $[0, 2 π]$ , while the magnitude is kept as $0.1$ of the canvas size. When the digit touches the boundary, the velocity is reflected.

Setup. We use the first 10 frames as $v_{1}$ (namely $k = 10$ ), and we construct $v_{2}$ by referring to the 20-th frame (namely $t = 20$ ). The CNN backbone consists of 4 layers with $8, 16, 32, 64$ filters from low to high. Average pooling is applied after the last convolutional layer, resulting in a 64 dimensional representation. The dimensions of the hidden layer and output in LSTM are both 64.

Training. We perform intra-batch contrast. Namely, inside each batch of size 128, we contrast each sample with the other 127 samples. We train for 200 epochs, with the learning rate initialized as $0.03$ and decayed with cosine annealing.

b.2 Spatial Patches with Different Distance

Setup and Training. The backbone network is a tiny AlexNet, following [30]. We train for $3000$ epochs, with the learning rate initialized as $0.03$ and decayed with cosine annealing.

Evaluation. We evaluate the learned representation on both STL-10 and CIFAR-10 datasets. For CIFAR-10, we resize the image to 64 $\times$ 64 to extract features. The linear classifier is trained for 100 epochs.

b.3 Channel Splitting with Various Color Spaces

Setup and Training. The backbone network is also a tiny AlexNet, with the modification of adapting the first layer to input of $1$ or $2$ channels. We follow the training recipe in [65].

Evaluation. For the evaluation on STL-10 dataset, we train a linear classifier for 100 epochs and report the single-crop classification accuracy. For NYU-Depth-v2 segmentation task, we freeze the backbone network and train a decoder on top of the learned representation. We report the mean IoU for labeled classes.

b.4 Frequency Separation

Setup and Training. The setup is almost the same as that in color channel splitting experiments, except that each view consists of three input channels. We follow the training recipe in [65].

Evaluation. We train a linear classifier for 100 epochs on STL-10 dataset and 40 epochs on TinyImageNet dataset.

b.5 Un-/Semi-supervised View Learning

Figure 13: Volume-preserving (a), and none volume-preserving (b) invertible model.

Invertible Generator. Figure 13 shows the basic building block for the Volume-Preserving (VP) and None-Volume-Preserving (NVP) invertible view generator. The $F$ and $G$ are pixel-wise convolutional function, i.e., convolutional layers with 1 $\times$ 1 kernel. $X_{1}$ and $Y_{1}$ represent a single channel of the input and output respectively, while $X_{2}$ and $Y_{2}$ represent the other two channels. While stacking basic building blocks, we alternatively select the first, second, and the third channel as $X_{1}$ , to enhance the expressivity of view generator.

Setup and Training. For unsupervised view learning that only uses the adversarial $I_{N C E}$ loss, we found the training is relatively unstable, as also observed in GAN [23]. We found the learning rate of view generator should be larger than that of $I_{N C E}$ approximator. Concretely, we use Adam optimizer [35], and we set the learning rates of view generator and $I_{N C E}$ approximator as $2 e$ - $4$ and $6 e$ - $4$ , respectively. For the semi-supervised view learning, we found the training is stable across different learning rate combinations, which we considered as an advantage. To be fair, we still use the same learning rates for both view generator and $I_{N C E}$ approximator.

Contrastive Learning and Evaluation. After the view learning stage, we perform contrastive learning and evaluation by following the recipe in B.3.

Appendix C Pascal VOC Object Detection

Setup. We strictly follow the setting introduced in [26]. Specifically, We use Faster R-CNN [56] with R50-C4 architecture. We fine-tune all layers with 24000 iterations, each consisting of 16 images.

pre-train	AP $_{50}$	AP	AP $_{75}$	ImageNet Acc(%)
random init.*	60.2	33.8	33.1	-
supervised*	81.3	53.5	58.8	76.1
InstDis	80.9	55.2	61.2	59.5
PIRL	81.0	55.5	61.3	61.7
MoCo*	81.5	55.9	62.6	60.6
InfoMin Aug. (ours)	82.7	57.6	64.6	70.1

Table 6: Pascal VOC object detection. All contrastive models are pretrained for 200 epochs on ImageNet for fair comparison. We use Faster R-CNN R50-C4 architecture for object detection. APs are reported using the average of 5 runs. * we use numbers from [26] since the setting is exactly the same.

Appendix D Instance Segmentation on COCO

We evaluated the transferability of various models pre-trained with InfoMin, under different detection frameworks and fine-tuning schedules. In all cases we tested, models pre-trained with InfoMin outperform those pre-trained with supervised cross-entropy loss. Interestingly, ResNeXt-152 trained with InfoMin on ImageNet-1K beats its supervised counterpart trained on ImageNet 5K, which is 6 $x$ times larger. Bounding box AP and mask Ap are reported on val2017

d.1 ResNet-50 with Mask R-CNN, C4 architecture

The results of Mask R-CNN with R-50 C4 backbone are shown in Table 7. We experimented with 1 $x$ and 2 $x$ schedule.

pre-train	AP $_{~{}}^{bb}$	AP $_{50}^{bb}$	AP $_{75}^{bb}$	AP $_{~{}}^{mk}$	AP $_{50}^{mk}$	AP $_{75}^{mk}$
random init*	26.4	44.0	27.8	29.3	46.9	30.8
supervised*	38.2	58.2	41.2	33.3	54.7	35.2
MoCo*	38.5( $↑$ 0.3)	58.3( $↑$ 0.1)	41.6( $↑$ 0.4)	33.6( $↑$ 0.1)	54.8( $↑$ 0.1)	35.6(

	$I (y; v_{1} \| z_{1})$	$= I (y; v_{1}) - I (y; v_{1}; z_{1})$
		$= I (y; v_{1}; v_{2}) + I (y; v_{1} \| v_{2}) - I (y; v_{1}; z_{1})$
		$= I (y; v_{1}; v_{2}) + I (y; v_{1} \| v_{2}) - [I (y; v_{1}; z_{1}; v_{2}) + I (y; v_{1}; z_{1} \| v_{2})]$
		$= I (y; v_{1} \| v_{2}) + [I (y; v_{1}; v_{2}) - I (y; v_{1}; z_{1}; v_{2})] - I (y; v_{1}; z_{1} \| v_{2})$
		$= I (y; v_{1} \| v_{2}) + I (y; v_{1}; v_{2} \| z_{1}) - I (y; v_{1}; z_{1} \| v_{2})$
		$= I (y; v_{1} \| v_{2}) + I (y; v_{1}; v_{2} \| z_{1}) + I (y; z_{1} \| v_{1}, v_{2})      0 - I (y; z_{1} \| v_{2})$

		$= I (y; v_{1} \| v_{2}) + I (v_{1}; v_{2} \| z_{1}) - I (v_{1}; v_{2} \| y, z_{1})      0$
		$= I (y; v_{1} \| v_{2}) + I (v_{1}; v_{2} \| z_{1})$

	$I (z_{1}; v_{1})$	$= I (z_{1}; v_{1} \| y) + I (z_{1}; v_{1}; y)$
		$= I (z_{1}; v_{1} \| y) + I (v_{1}; y) - I (v_{1}; y \| z_{1})      0$
		$= I (z_{1}; v_{1} \| y) + I (v_{1}; y)$
		$\geq I (v_{1}; y)$

What makes for good views for contrastive learning

Authors

Rethinking Pre-training and Self-training

Unsupervised pre-training helps to conserve views from input distribution

Contrastive Multiview Coding

DetCo: Unsupervised Contrastive Learning for Object Detection

Self-supervised Pre-training with Hard Examples Improves Visual Representations

Contrastive Pre-training for Sequential Recommendation

Learning Representations by Maximizing Mutual Information Across Views

This week in AI

1 Introduction

2 Related Work

3 Preliminary: Contrastive Representation Learning

4 What Are the Optimal Views for Contrastive Learning?

4.1 InfoMin Principle: Views that Only Share Label Information

Proposition 4.1.

4.2 A Toy Example: Colorful Moving-MNIST

5 A Sweet Spot in Mutual Information: Reverse-U Shape

5.1 Three Regimes of Information Captured

5.2 Practical Cases of Reducing I(v1;v2)

5.2.1 Reducing I(v1;v2) with Spatial Distance

5.2.2 Reducing I(v1;v2) with Different Color Spaces

5.2.3 Reducing I(v1;v2) with Frequency Separation

6 Synthesizing Effective Views

6.1 Random Views

6.2 Unsupervised: Minimize I(v1;v2)

6.3 Semi-supervised View Learning: Find Views that Share the Label Information

7 Data Augmentation as InfoMin

7.1 A Unified View of Recent State-of-the-Art Methods

7.2 Analysis of Data Augmentation as it relates to MI and Transfer Performance

7.3 Results on ImageNet Benchmark

7.4 Transferring Representations

8 Conclusion

Acknowledgments.

References

Appendix A Proof of Proposition 1

a.1 Properties of MI [12]:

a.2 Proof

Proposition A.1.

Proof.

Proposition A.2.

Proof.

Proposition A.3.

Proof.

Appendix B Implementation Details

b.1 Colorful Moving MNIST

b.2 Spatial Patches with Different Distance

b.3 Channel Splitting with Various Color Spaces

b.4 Frequency Separation

b.5 Un-/Semi-supervised View Learning

Appendix C Pascal VOC Object Detection

Appendix D Instance Segmentation on COCO

d.1 ResNet-50 with Mask R-CNN, C4 architecture

5.2 Practical Cases of Reducing $I (v_{1}; v_{2})$

5.2.1 Reducing $I (v_{1}; v_{2})$ with Spatial Distance

5.2.2 Reducing $I (v_{1}; v_{2})$ with Different Color Spaces

5.2.3 Reducing $I (v_{1}; v_{2})$ with Frequency Separation

6.2 Unsupervised: Minimize $I (v_{1}; v_{2})$