Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

06/13/2020 ∙ by Jean-Bastien Grill, et al. ∙ 0

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods intrinsically rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches 74.3% top-1 classification accuracy on ImageNet using the standard linear evaluation protocol with a ResNet-50 architecture and 79.6% with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

[1, 2, 3, 4, 5, 6, 7]

Figure 1: Performance of BYOL on ImageNet (linear evaluation) using ResNet-50 and our best architecture ResNet-200 (), compared to other unsupervised and supervised (Sup.) baselines [8].

Learning good image representations is a key challenge in computer vision 

[1, 2, 3] as it allows for efficient training on downstream tasks [4, 5, 6, 7]. Many different training approaches have been proposed to learn such representations, usually relying on visual pretext tasks. Among them, state-of-the-art contrastive methods [8, 9, 10, 11, 12] are trained by reducing the distance between representations of different augmented views of the same image (‘positive pairs’), and increasing the distance between representations of augmented views from different images (‘negative pairs’). These methods require careful treatment of negative pairs [13] by either relying on large batch sizes [8, 12], memory banks [9] or customized mining strategies [14, 15] to retrieve the negative pairs. In addition, their performance critically depends on the choice of image augmentations [8, 12].

In this paper, we introduce Bootstrap Your Own Latent (BYOL), a new algorithm for self-supervised learning of image representations. BYOL achieves higher performance than state-of-the-art contrastive methods without using negative pairs. It iteratively bootstraps111Throughout this paper, the term bootstrap is used in its idiomatic sense rather than the statistical sense. the outputs of a network to serve as targets for an enhanced representation. Moreover, BYOL is more robust to the choice of image augmentations than contrastive methods; we suspect that not relying on negative pairs is one of the leading reasons for its improved robustness. While previous methods based on bootstrapping have used pseudo-labels or cluster indices [16, 17] as targets, we propose to directly bootstrap the representations. In particular, BYOL uses two neural networks, referred to as online and target networks, that interact and learn from each other. Starting from an augmented view of an image, BYOL trains its online network to predict the target network’s representation of another augmented view of the same image. Even if this problem admits trivial solutions, e.g., outputting zero for all images, we empirically show that using a slow-moving average of the online network as the target network suffices to avoid such collapse to uninformative solutions.

We evaluate the representation learned by BYOL on ImageNet [18] and other vision benchmarks using ResNet architectures [19]

. Under the linear evaluation protocol on ImageNet, consisting in training a linear classifier on top of the frozen representation, BYOL reaches

top-1 accuracy with a standard ResNet- and top-1 accuracy with a larger ResNet (Figure 1). In the semi-supervised and transfer settings on ImageNet, we obtain results on par or superior to the current state of the art. Our contributions are: (i) We introduce BYOL, a self-supervised representation learning method (Section 3) which achieves state-of-the-art results under the linear evaluation protocol on ImageNet without using negative pairs. (ii) We show that our learned representation outperforms the state of the art on semi-supervised and transfer benchmarks (Section 4). (iii) We show that BYOL is more resilient to changes in the batch size and in the set of image augmentations compared to its contrastive counterparts (Section 5). In particular, BYOL suffers a much smaller performance drop than SimCLR, a strong contrastive baseline, when only using random crops as image augmentations.

2 Related work

Most unsupervised methods for representation learning can be categorized as either generative or discriminative [20, 8]. Generative approaches to representation learning build a distribution over data and latent embedding and use the learned embeddings as image representations. Many of these approaches rely either on auto-encoding of images [21, 22, 23] or on adversarial learning [24], jointly modelling data and representation [25, 26, 27, 28]. Generative methods typically operate directly in pixel space. This however is computationally expensive, and the high level of detail required for image generation may not be necessary for representation learning.

Among discriminative methods, contrastive methods [9, 10, 29, 30, 31, 11, 32, 33] currently achieve state-of-the-art performance in self-supervised learning [34, 8, 12]. Contrastive approaches avoid a costly generation step in pixel space by bringing representation of different views of the same image closer (‘positive pairs’), and spreading representations of views from different images (‘negative pairs’) apart [35, 36]. Contrastive methods often require comparing each example with many other examples to work well [9, 8] prompting the question of whether using negative pairs is necessary.

DeepCluster [17] partially answers this question. It uses bootstrapping on previous versions of its representation to produce targets for the next representation; it clusters data points using the prior representation, and uses the cluster index of each sample as a classification target for the new representation. While avoiding the use of negative pairs, this requires a costly clustering phase and specific precautions to avoid collapsing to trivial solutions.

Some self-supervised methods are not contrastive but rely on using auxiliary handcrafted prediction tasks to learn their representation. In particular, relative patch prediction [20, 36], colorizing gray-scale images [37, 38]

, image inpainting 

[39], image jigsaw puzzle [40]

, image super-resolution 

[41], and geometric transformations [42, 43] have been shown to be useful. Yet, even with suitable architectures [44], these methods are being outperformed by contrastive methods [34, 8, 12].

Our approach has some similarities with Predictions of Bootstrapped Latents (PBL, [45]

), a self-supervised representation learning technique for reinforcement learning. PBL jointly trains the agent’s history representation and an encoding of future observations. The observation encoding is used as a target to train the agent’s representation, and the agent’s representation as a target to train the observation encoding. Unlike PBL, BYOL uses a slow-moving average of its representation to provide its targets, and does not require a second network.

In self-supervised learning, MoCo [9] uses a moving average network (momentum encoder) to maintain consistent representations of negative pairs drawn from a memory bank. Instead, BYOL uses a moving average network to produce prediction targets as a means of stabilizing the bootstrap step. We show in Section 5 that this mere stabilizing effect can also improve existing contrastive methods.

3 Method

We start by motivating our method before explaining its details in Section 3.1. Many successful self-supervised learning approaches build upon the cross-view prediction framework introduced in [46]. Typically, these approaches learn representations by predicting different views (e.g., different random crops) of the same image from one another. Many such approaches cast the prediction problem directly in representation space: the representation of an augmented view of an image should be predictive of the representation of another augmented view of the same image. However, predicting directly in representation space can lead to collapsed representations: for instance, a representation that is constant across views is always fully predictive of itself. Contrastive methods circumvent this problem by reformulating the prediction problem into one of discrimination: from the representation of an augmented view, they learn to discriminate between the representation of another augmented view of the same image, and the representations of augmented views of different images. In the vast majority of cases, this prevents the training from finding collapsed representations. Yet, this discriminative approach typically requires comparing each representation of an augmented view with many negative examples, to find ones sufficiently close to make the discrimination task challenging. In this work, we thus tasked ourselves to find out whether these negative examples are indispensable to prevent collapsing while preserving high performance.

To prevent collapse, a straightforward solution is to use a fixed randomly initialized network to produce the targets for our predictions. While avoiding collapse, it empirically does not result in very good representations. Nonetheless, it is interesting to note that the representation obtained using this procedure can already be much better than the initial fixed representation. In our ablation study (Section 5), we apply this procedure by predicting a fixed randomly initialized network and achieve top-1 accuracy (Table 4(a)) on the linear evaluation protocol on ImageNet, whereas the randomly initialized network only achieves by itself. This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation. From there, we can expect to build a sequence of representations of increasing quality by iterating this procedure, using subsequent online networks as new target networks for further training. In practice, BYOL generalizes this bootstrapping procedure by iteratively refining its representation, but using a slowly moving exponential average of the online network as the target network instead of fixed checkpoints.222Target networks have been introduced in [47] and are commonly used [48, 49, 50] in deep reinforcement learning (RL). In deep RL, target networks stabilize the bootstrapping updates provided by the Bellman equation, making them appealing to stabilize the bootstrap mechanism in BYOL. While fixed target networks are more common in deep RL, BYOL uses a weighted moving average of previous networks, similar to [51], in order to provide smoother changes in the target representation.

3.1 Description of BYOL

Figure 2: BYOL’s architecture. BYOL minimizes a similarity loss between and , where are the trained weights, are an exponential moving average of and means stop-gradient. At the end of training, everything but is discarded and is used as the image representation.

BYOL’s goal is to learn a representation which can then be used for downstream tasks. As described previously, BYOL uses two neural networks to learn: the online and target networks. The online network is defined by a set of weights and is comprised of three stages: an encoder , a projector  and a predictor , as shown in Figure 2. The target network has the same architecture as the online network, but uses a different set of weights . The target network provides the regression targets to train the online network, and its parameters are an exponential moving average of the online parameters  [51]. More precisely, given a target decay rate , after each training step we perform the following update,

(1)

Given a set of images , an image sampled uniformly from , and two distributions of image augmentations and , BYOL produces two augmented views and from by applying respectively image augmentations and . From the first augmented view , the online network outputs a representation and a projection . The target network outputs and the target projection from the second augmented view . We then output a prediction of and -normalize both and to and . Finally we define the following mean squared error between the normalized predictions and target projections,333While we could directly predict the representation  and not a projection , previous work [8] have empirically shown that using this projection improves performance.

(2)

We symmetrize the loss in Equation 2 by separately feeding to the online network and to the target network to compute . At each training step, we perform a stochastic optimization step to minimize with respect to only, but not , as depicted by the stop gradient in Figure 2. At the end of training, we only keep the encoder ; as in [9]. When comparing to other methods we consider the number of inference-time weights only in the final representation . The full training procedure is summarized in Appendix A, and python pseudo-code based on the libraries JAX [52] and Haiku [53] is provided in Appendix G.

3.2 Implementation details

Image augmentations

BYOL uses the same set of image augmentations as in SimCLR [8]. First, a random patch of the image is selected and resized to with a random horizontal flip, followed by a color distortion, consisting of a random sequence of brightness, contrast, saturation, hue adjustments, and an optional grayscale conversion. Finally Gaussian blur and solarization are applied to the patches. Additional details on the image augmentations are in Appendix B.

Architecture

We use a convolutional residual network [19] with 50 layers and post-activation (ResNet- v1) as our base parametric encoders and . We also use deeper (, , and layers) and wider (from to ) ResNets, as in [54, 44, 8]. Specifically, the representation corresponds to the output of the final average pooling layer, which has a feature dimension of (for a width multiplier of ). As in SimCLR [8], the representation  is projected to a smaller space by a

multi-layer perceptron

(MLP) , and similarly for the target projection . This MLP consists in a linear layer with output size

followed by batch normalization 

[55]

, rectified linear units (ReLU

[56], and a final linear layer with output dimension . Contrary to SimCLR, the output of this MLP is not batch normalized. The predictor uses the same architecture as .

Optimization

We use the LARS optimizer [57] with a cosine decay learning rate schedule [58], without restarts, over epochs, with a warm-up period of epochs. We set the base learning rate to scaled linearly [59] with the batch size (). In addition, we use a global weight decay parameter of while excluding the biases and batch normalization parameters from both LARS adaptation and weight decay. For the target network, the exponential moving average parameter starts from and is increased to one during training. Specifically, we set with the current training step and the maximum number of training steps. We use a batch size of split over Cloud TPU v cores. With this setup, training takes approximately hours for a ResNet-

. All hyperparameters are summarized in Appendix G.

4 Experimental evaluation

We assess the performance of BYOL’s representation444Our training code and pre-trained models will be publicly released at a later date. after self-supervised pretraining on the training set of the ImageNet ILSVRC-2012 dataset [18]

. We first evaluate it on ImageNet (IN) in both linear evaluation and semi-supervised setups. We then measure its transfer capabilities on other datasets and tasks, including classification, segmentation, object detection and depth estimation. For comparison, we also report scores for a representation trained using labels from the train ImageNet subset, referred to as Supervised-IN. In

Appendix E, we assess the generality of BYOL by pretraining a representation on the Places365-Standard dataset [60] before reproducing this evaluation protocol.

Linear evaluation on ImageNet

We first evaluate BYOL’s representation by training a linear classifier on top of the frozen representation, following the procedure described in [44, 61, 37, 10, 8], and section C.1; we report top- and top- accuracies in % on the test set in Table 1. With a standard ResNet- () BYOL obtains top- accuracy ( top- accuracy), which is a (resp. ) improvement over the previous self-supervised state of the art [12]. This tightens the gap with respect to the supervised baseline of [8], , but is still significantly below the stronger supervised baseline of [62], . With deeper and wider architectures, BYOL consistently outperforms the previous state of the art (Section C.2), and obtains a best performance of top- accuracy, ranking higher than previous self-supervised approaches. On a ResNet- () BYOL achieves , similar to the of the best supervised baseline in [8] for the same architecture.

Method Top- Top-
Local Agg. -
PIRL [32] -
CPC v2 [29]
CMC [11]
SimCLR [8]
MoCo v2 [34] -
InfoMin Aug. [12]
BYOL (ours)
(a) ResNet-50 encoder.
Method Architecture Param. Top- Top-
SimCLR [8] ResNet- () M
CMC [11] ResNet- () M
BYOL (ours) ResNet- () M
CPC v2 [29] ResNet- M
MoCo [9] ResNet- () M -
SimCLR [8] ResNet- () M
BYOL (ours) ResNet- () M
BYOL (ours) ResNet- () M
(b) Other ResNet encoder architectures.
Table 1: Top-1 and top-5 accuracies (in %) under linear evaluation on ImageNet.

Semi-supervised training on ImageNet

Next, we evaluate the performance obtained when fine-tuning BYOL’s representation on a classification task with a small subset of ImageNet’s train set, this time using label information. We follow the semi-supervised protocol of [61, 63, 8, 29] detailed in Section C.1, and use the same fixed splits of respectively and of ImageNet labeled training data as in [8]. We report both top- and top- accuracies on the test set in Table 2. BYOL consistently outperforms previous approaches across a wide range of architectures. Additionally, as detailed in Section C.1, BYOL reaches top- accuracy with ResNet-50 when fine-tuning over of ImageNet labels.

Method Top- Top-
Supervised [64]
InstDisc - -
PIRL [32] - -
SimCLR [8]
BYOL (ours)
(a) ResNet-50 encoder.
Method Architecture Param. Top- Top-
CPC v2 [29] ResNet- M - -
SimCLR [8] ResNet- () M
BYOL (ours) ResNet- () M
SimCLR [8] ResNet- () M
BYOL (ours) ResNet- () M
BYOL (ours) ResNet- () M
(b) Other ResNet encoder architectures.
Table 2: Semi-supervised training with a fraction of ImageNet labels.

Transfer to other classification tasks

We evaluate our representation on other classification datasets to assess whether the features learned on ImageNet (IN) are generic and thus useful across image domains, or if they are ImageNet-specific. We perform linear evaluation and fine-tuning on the same set of classification tasks used in [8, 61], and carefully follow their evaluation protocol, as detailed in Appendix D. Performance is reported using standard metrics for each benchmark, and results are provided on a held-out test set after hyperparameter selection on a validation set. We report results in Table 3, both for linear evaluation and fine-tuning. BYOL outperforms SimCLR on all benchmarks and the Supervised-IN baseline on of the benchmarks, providing only slightly worse performance on the remaining benchmarks. BYOL’s representation can be transferred over to small images, e.g., CIFAR [65], landscapes, e.g., SUN397 [66] or VOC2007 [67], and textures, e.g., DTD [68].

Method Food101 CIFAR10 CIFAR100 Birdsnap SUN397 Cars Aircraft VOC2007 DTD Pets Caltech-101 Flowers
Linear evaluation:
BYOL (ours)
SimCLR (repro)
SimCLR [8]
Supervised-IN [8]
Fine-tuned:
BYOL (ours)
SimCLR (repro)
SimCLR [8]
Supervised-IN [8]
Random init [8]
Table 3: Transfer learning results from ImageNet (IN) with the standard ResNet-50 architecture.

Transfer to other vision tasks

We evaluate our representation on different tasks relevant to computer vision practitioners, namely semantic segmentation, object detection and depth estimation. With this evaluation, we assess whether BYOL’s representation generalizes beyond classification tasks.

We first evaluate BYOL on the VOC2012 semantic segmentation task as detailed in Section D.4, where the goal is to classify each pixel in the image [7]. We report the results in Table 3(a). BYOL outperforms both the Supervised-IN baseline ( mIoU) and SimCLR ( mIoU).

Similarly, we evaluate on object detection by reproducing the setup in [9] using a Faster R-CNN architecture [69], as detailed in Section D.5. We fine-tune on trainval2007 and report results on test2007 using the standard AP metric; BYOL is significantly better than the Supervised-IN baseline ( AP) and SimCLR ( AP).

Finally, we evaluate on depth estimation on the NYU v2 dataset, where the depth map of a scene is estimated given a single RGB image. Depth prediction measures how well a network represents geometry, and how well that information can be localized to pixel accuracy [36]. The setup is based on [70] and detailed in Section D.6. We evaluate on the commonly used test subset of images and report results using several common metrics in Table 3(b): relative (rel) error, root mean squared (rms) error, and the percent of pixels (pct) where the error, , is below thresholds where is the predicted depth and is the ground truth depth [36]. BYOL is better or on par with other methods for each metric. For instance, the challenging pct.  measure is respectively improved by points and points compared to supervised and SimCLR baselines.

Method AP mIoU
Supervised-IN [9]
MoCo [9]
SimCLR (repro)
BYOL (ours)
(a) Transfer results in semantic segmentation and object detection.
Higher better Lower better
Method pct.  pct.  pct.  rms rel
Supervised-IN [70]
SimCLR (repro)
BYOL (ours)
(b) Transfer results on NYU v2 depth estimation.
Table 4: Results on transferring BYOL’s representation to other vision tasks.

5 Building intuitions with ablations

We present ablations on BYOL to give an intuition of its behavior and performance. For reproducibility, we run each configuration of parameters over three seeds, and report the average performance. We also report the half difference between the best and worst runs when it is larger than . Although previous works perform ablations at epochs [8, 12], we notice that relative improvements at epochs do not always hold over longer training. For this reason, we run ablations over epochs on TPU v cores, which yields consistent results compared to our baseline training of epochs. For all the experiments in this section, we set the initial learning rate to with batch size , the weight decay to as in SimCLR [8] and the base target decay rate to . In this section we report results in top- accuracy on ImageNet under the linear evaluation protocol as in Section C.1.

Batch size

Among contrastive methods, the ones that draw negative examples from the minibatch suffer performance drops when their batch size is reduced. BYOL does not use negative examples and we expect it to be more robust to smaller batch sizes. To empirically verify this hypothesis, we train both BYOL and SimCLR using different batch sizes from to . To avoid re-tuning other hyperparameters, we average gradients over consecutive steps before updating the online network when reducing the batch size by a factor . The target network is updated once every steps, after the update of the online network; we accumulate the -steps in parallel in our runs.

As shown in Figure 2(a), the performance of SimCLR rapidly deteriorates with batch size, likely due to the decrease in the number of negative examples. In contrast, the performance of BYOL remains stable over a wide range of batch sizes from to , and only drops for smaller values due to batch normalization layers in the encoder.555The only dependency on batch size in our training pipeline sits within the batch normalization layers.

(a) Impact of batch size
(b) Impact of progressively removing transformations
Figure 3: Decrease in top- accuracy (in % points) of BYOL and our own reproduction of SimCLR at epochs, under linear evaluation on ImageNet.

Image augmentations

Contrastive methods are sensitive to the choice of image augmentations. For instance, SimCLR does not work well when removing color distortion from its image augmentations. As an explanation, SimCLR shows that crops of the same image mostly share their color histograms. At the same time, color histograms vary across images. Therefore, when a contrastive task only relies on random crops as image augmentations, it can be mostly solved by focusing on color histograms alone. As a result the representation is not incentivized to retain information beyond color histograms. To prevent that, SimCLR adds color distortion to its set of image augmentations. Instead, BYOL is incentivized to keep any information captured by the target representation into its online network, to improve its predictions. Therefore, even if augmented views of a same image share the same color histogram, BYOL is still incentivized to retain additional features in its representation. For that reason, we believe that BYOL is more robust to the choice of image augmentations than contrastive methods.

Results presented in Figure 2(b) support this hypothesis: the performance of BYOL is much less affected than the performance of SimCLR when removing color distortions from the set of image augmentations ( accuracy points for BYOL, accuracy points for SimCLR). When image augmentations are reduced to mere random crops, BYOL still displays good performance (, i.e. points from ), while SimCLR loses more than a third of its performance (, i.e. points from ). We report additional ablations in Section F.3.

Bootstrapping

BYOL uses the projected representation of a target network, whose weights are an exponential moving average of the weights of the online network, as target for its predictions. This way, the weights of the target network represent a delayed and more stable version of the weights of the online network. When the target decay rate is , the target network is never updated, and remains at a constant value corresponding to its initialization. When the target decay rate is , the target network is instantaneously updated to the online network at each step. There is a trade-off between updating the targets too often and updating them too slowly, as illustrated in Table 4(a). Instantaneously updating the target network () destabilizes training, yielding very poor performance while never updating the target () makes the training stable but prevents iterative improvement, ending with low-quality final representation. All values of the decay rate between and yield performance above top- accuracy at epochs.

Target Top-
Constant random network
Moving average of online
Moving average of online
Moving average of online
Stop gradient of online
(a) Results for different target modes. In the stop gradient of online, is kept constant throughout training.
Method Predictor Target network Top-
BYOL 0
1
1
SimCLR 1
1
0
0
0
(b) Intermediate variants between BYOL and SimCLR.
Table 5: Ablations with top- accuracy (in %) at epochs under linear evaluation on ImageNet.

Ablation to contrastive methods

In this subsection, we recast SimCLR and BYOL using the same formalism to better understand where the improvement of BYOL over SimCLR comes from. Let us consider the following objective that extends the InfoNCE objective [10],

(3)

where is a fixed temperature, a weighting coefficient, the batch size, and are batches of augmented views where for any batch index , and are augmented views from the same image; the real-valued function quantifies pairwise similarity between augmented views. For any augmented view  we denote and . For given and , we consider the normalized dot product

(4)

Up to minor details (cf. Section F.5), we recover the SimCLR loss with (no predictor), (no target network) and . We recover the BYOL loss when using a predictor and a target network, i.e., and with . To evaluate the influence of the target network, the predictor and the coefficient , we perform an ablation over them. Results are presented in Table 4(b) and more details are given in Section F.4.

The only variant that performs well without negative examples (i.e., with ) is BYOL, using both a bootstrap target network and a predictor. Adding the negative pairs to BYOL’s loss without re-tuning the temperature parameter hurts its performance. In Section F.4, we show that we can add back negative pairs and still match the performance of BYOL with proper tuning of the temperature.

Simply adding a target network to SimCLR already improves performance ( points). This sheds new light on the use of the target network in MoCo [9], where the target network is used to provide more negative examples. Here, we show that by mere stabilization effect, even when using the same number of negative examples, using a target network is beneficial. Finally, we observe that modifying the architecture of to include a predictor only mildly affects the performance of SimCLR.

6 Conclusion

We introduced BYOL, a new algorithm for self-supervised learning of image representations. BYOL learns its representation by predicting previous versions of its outputs, without using negative pairs. We show that BYOL achieves state-of-the-art results on various benchmarks. In particular, under the linear evaluation protocol on ImageNet with a ResNet- (), BYOL achieves a new state of the art and bridges most of the remaining gap between self-supervised methods and the supervised learning baseline of [8]. Using a ResNet- , BYOL reaches a top- accuracy of which improves over the previous state of the art () while using fewer parameters.

Nevertheless, BYOL remains dependent on existing sets of augmentations that are specific to vision applications. To generalize BYOL to other modalities (e.g., audio, video, text, …) it is necessary to obtain similarly suitable augmentations for each of them. Designing such augmentations may require significant effort and expertise. Therefore, automating the search for these augmentations would be an important next step to generalize BYOL to other modalities.

Broader impact

The presented research should be categorized as research in the field of unsupervised learning. This work may inspire new algorithms, theoretical, and experimental investigation. The algorithm presented here can be used for many different vision applications and a particular use may have both positive or negative impacts, which is known as the dual use problem. Besides, as vision datasets could be biased, the representation learned by BYOL could be susceptible to replicate these biases.

Acknowledgements

The authors would like to thank the following people for their help throughout the process of writing this paper, in alphabetical order: Aaron van den Oord, Andrew Brock, Jason Ramapuram, Jeffrey De Fauw, Karen Simonyan, Katrina McKinney, Nathalie Beauguerlange, Olivier Henaff, Oriol Vinyals, Pauline Luc, Razvan Pascanu, Sander Dieleman, and the DeepMind team.

References

Appendix A Algorithm

Inputs : 
, and set of images and distributions of transformations
, , and initial online parameters, encoder, projector, and predictor
, , initial target parameters, target encoder, and target projector
optimizer, updates online parameters using the loss gradient
and total number of optimization steps and batch size
and target network update schedule and learning rate schedule
1 for  to  do
         // sample a batch of images
2       for  do
               // sample image transformations
             and   // compute projections
             and   // compute target projections
               // compute the loss for
3            
4       end for
         // compute the total loss gradient w.r.t. 
         // update online parameters
         // update target parameters
5      
6 end for
Output : encoder
Algorithm 1 BYOL: Bootstrap Your Own Latent

Appendix B Image augmentations

During self-supervised training, BYOL uses the following image augmentations (which are a subset of the ones presented in [8]):

  • random cropping: a random patch of the image is selected, with an area uniformly sampled between 8% and 100% of that of the original image, and an aspect ratio logarithmically sampled between and . This patch is then resized to the target size of

    using bicubic interpolation;

  • optional left-right flip;

  • color jittering: the brightness, contrast, saturation and hue of the image are shifted by a uniformly random offset applied on all the pixels of the same image. The order in which these shifts are performed is randomly selected for each patch;

  • color dropping: an optional conversion to grayscale. When applied, output intensity for a pixel corresponds to its luma component, computed as ;

  • Gaussian blurring: for a image, a square Gaussian kernel of size

    is used, with a standard deviation uniformly sampled over

    ;

  • solarization: an optional color transformation for pixels with values in .

Augmentations from the sets and  (introduced in Section 3

) are compositions of the above image augmentations in the listed order, each applied with a predetermined probability. The image augmentations parameters are listed in

Table 6.

During evaluation, we use a center crop similar to [8]: images are resized to pixels along the shorter side using bicubic resampling, after which a center crop is applied. In both training and evaluation, we normalize color channels by subtracting the average color and dividing by the standard deviation, computed on ImageNet, after applying the augmentations.

Parameter
Random crop probability
Flip probability
Color jittering probability
Brightness adjustment max intensity
Contrast adjustment max intensity
Saturation adjustment max intensity
Hue adjustment max intensity
Color dropping probability
Gaussian blurring probability
Solarization probability
Table 6: Parameters used to generate image augmentations.

Appendix C Evaluation on ImageNet training

c.1 Self-supervised learning evaluation on ImageNet

Linear evaluation protocol on ImageNet

As in [44, 61, 8, 34], we use the standard linear evaluation protocol on ImageNet, which consists in training a linear classifier on top of the frozen representation, i.e., without updating the network parameters nor the batch statistics. At training time, we apply spatial augmentations, i.e., random crops with resize to pixels, and random flips. At test time, images are resized to pixels along the shorter side using bicubic resampling, after which a

center crop is applied. In both cases, we normalize the color channels by subtracting the average color and dividing by the standard deviation (computed on ImageNet), after applying the augmentations. We optimize the cross-entropy loss using SGD with Nesterov momentum over

epochs, using a batch size of and a momentum of

. We do not use any regularization methods such as weight decay, gradient clipping 

[71], tclip [31]

, or logits regularization. We finally sweep over

learning rates on a local validation set (10009 images from ImageNet train set), and report the accuracy of the best validation hyperparameter on the test set (which is the public validation set of the original ILSVRC2012 ImageNet dataset).

Variant on linear evaluation on ImageNet

In this paragraph only, we deviate from the protocol of [8, 34] and propose another way of performing linear evaluation on top of a frozen representation. This method achieves better performance both in top-1 and top-5 accuracy.

  • [leftmargin=*]

  • We replace the spatial augmentations (random crops with resize to pixels and random flips) with the pre-train augmentations of Appendix B. This method was already used in [29] with a different subset of pre-train augmentations.

  • We regularize the linear classifier as in [31]666https://github.com/Philip-Bachman/amdim-public/blob/master/costs.py by clipping the logits using a hyperbolic tangent function

    where is a positive scalar, and by adding a logit-regularization penalty term in the loss

    where are the logits, are the target labels, and is the regularization parameter. We set and .

We report in Table 7 the top-1 and top-5 accuracy on ImageNet using this modified protocol. These modifications in the evaluation protocol increase the BYOL’s top- accuracy from to with a ResNet- ().

Architecture Pre-train augmentations Logits regularization Top- Top-
ResNet- ()
ResNet- ()
ResNet- ()
Table 7: Different linear evaluation protocols on ResNet architectures by either replacing the spatial augmentations with pre-train augmentations, or regularizing the linear classifier. No pre-train augmentations and no logits regularization correspond to the evaluation protocol of the main paper, which is the same as in [8, 34].

Semi-supervised learning on ImageNet

We follow the semi-supervised learning protocol of [8, 64]. We first initialize the network with the parameters of the pretrained representation, and finetune it with a subset of ImageNet labels. At training time, we apply spatial augmentations, i.e., random crops with resize to pixels and random flips. At test time, images are resized to pixels along the shorter side using bicubic resampling, after which a center crop is applied. In both cases, we normalize the color channels by subtracting the average color and dividing by the standard deviation (computed on ImageNet), after applying the augmentations. We optimize the cross-entropy loss using SGD with Nesterov momentum. We used a batch size of , a momentum of . We do not use any regularization methods such as weight decay, gradient clipping [71], tclip [31], or logits rescaling. We sweep over the learning rate and the number of epochs and select the hyperparameters achieving the best performance on our local validation set to report test performance.

(a) Top-1 accuracy
(b) Top-5 accuracy
Figure 4: Semi-supervised training with a fraction of ImageNet labels on a ResNet- ().
Supervised: Semi-supervised ():
Method Top- Top- Method Top- Top-
Supervised[8] SimCLR [8]
AutoAugment [72] SimCLR (repro)
MaxUp [62] BYOL
Table 8: Semi-supervised training with the full ImageNet on a ResNet- (). We also report other fully supervised methods for extensive comparisons.

In Table 2 presented in the main text, we finetune the representation over the % and % ImageNet splits from [8] with various ResNet architectures.

In Figure 4, we finetune the representation over %, %, %, %, %, %, and % of the ImageNet dataset as in [29] with a ResNet- () architecture. In this case and contrary to Table 2 we don’t reuse the splits from SimCLR but we create our own via a balanced selection.

Finally, we finetune the representation over the full ImageNet dataset. We report the results in Table 8 along with supervised baselines trained on ImageNet. We observe that finetuning the SimCLR checkpoint does not yield better results (in our reproduction, which matches the results reported in the original paper [8]) than using a random initialization ( top-). Instead, BYOL’s initialization checkpoint leads to a high final score ( top-), higher than the vanilla supervised baseline of [8], matching the strong supervised baseline of AutoAugment[72] but still points below the stronger supervised baseline [62], which uses advanced supervised learning techniques.

c.2 Linear evaluation on larger architectures and supervised baselines

BYOL Supervised (ours) Supervised [8]
Architecture Multiplier Weights Top-1 Top-5 Top-1 Top-5 Top-1
ResNet-50 24M
ResNet-101 43M -
ResNet-152 58M -
ResNet-200 63M -
ResNet-50 94M
ResNet-101 170M -
ResNet-50 211M -
ResNet-152 232M -
ResNet-200 250M -
ResNet-50 375M
ResNet-101 382M -
ResNet-152 522M -
Table 9: Linear evaluation of BYOL on ImageNet using larger encoders. Top- and top- accuracies are reported in %.

Here we investigate the performance of BYOL with deeper and wider ResNet architectures. We compare ourselves to the best supervised baselines from [8] when available (rightmost column in table 9), which are also presented in Figure 1. Importantly, we close in on those baselines using the ResNet- () and the ResNet- () architectures, where we are within accuracy points of the supervised performance. To the best of our knowledge, this is the first time that the gap to supervised has been closed to such an extent using a self-supervised method under the linear evaluation protocol. Therefore, in order to ensure fair comparison, and suspecting that the supervised baselines’ performance in [8] could be even further improved with appropriate data augmentations, we also report on our own reproduction of strong supervised baselines. We use RandAugment [72] data augmentation for all large ResNet architectures (which are all version , as per [19]). We train our supervised baselines for up to epochs, using SGD with a Nesterov momentum value of , a cosine-annealed learning rate after a epochs linear warmup period, weight decay with a value of , and a label smoothing [73] value of . Results are presented in Figure 5.

Figure 5: Results for linear evaluation of BYOL compared to fully supervised baselines with various ResNet architectures.

Appendix D Transfer to other datasets

d.1 Datasets

Dataset Classes Original train examples Train examples Valid. examples Test examples Accuracy measure Test provided
ImageNet [18] Top-1 accuracy -
Food101 [74] Top-1 accuracy -
CIFAR-10 [65] Top-1 accuracy -
CIFAR-100 [65] Top-1 accuracy -
Birdsnap [75] Top-1 accuracy -
Sun397 (split 1) [66] Top-1 accuracy -
Cars [76] Top-1 accuracy -
Aircraft [77] Mean per-class accuracy Yes
PASCAL-VOC2007  [67] 11-point mAP / AP50 -
PASCAL-VOC2012  [67] Mean IoU -
DTD (split 1)  [68] Top-1 accuracy Yes
Pets [78] Mean per-class accuracy -
Caltech-101 [79] Mean per-class accuracy -
Places365 [60] Top-1 accuracy -
Flowers [80] Mean per-class accuracy Yes
Table 10: Characteristics of image datasets used in transfer learning. When an official test split with labels is not publicly available, we use the official validation split as test set, and create a held-out validation set from the training examples.

We perform transfer via linear classification and fine-tuning on the same set of datasets as in [8], namely Food- dataset [74], CIFAR- [65] and CIFAR- [65], Birdsnap [75], the SUN scene dataset [66], Stanford Cars [76], FGVC Aircraft [77], the PASCAL VOC classification task [67], the Describable Textures Dataset (DTD) [68], Oxford-IIIT Pets [78], Caltech- [79], and Oxford Flowers [80]. As in [8], we used the validation sets specified by the dataset creators to select hyperparameters for FGVC Aircraft, PASCAL VOC 2007, DTD, and Oxford 102 Flowers. On other datasets, we use the validation examples as test set, and hold out a subset of the training examples that we use as validation set. We use standard metrics for each datasets:

  • Top-: We compute the proportion of correctly classified examples.

  • Mean per class: We compute the top- accuracy for each class separately and then compute the empirical mean over the classes.

  • Point 11-mAP: We compute the empirical mean average precision as defined in [67].

  • Mean IoU: We compute the empirical mean Intersection-Over-Union as defined in [67].

  • AP50: We compute the Average Precision as defined in [67].

We detail the validation procedures for some specific datasets:

  • For Sun397 [66], the original dataset specifies 10 train/test splits, all of which contain 50 examples/images of 397 different classes. We use the first train/test split. The original dataset specifies no validation split and therefore, the training images have been further subdivided into 40 images per class for the train split and 10 images per class for the valid split.

  • For Birdsnap [75], we use a random selection of valid images with the same number of images per category as the test split.

  • For DTD [68], the original dataset specifies 10 train/validation/test splits, we only use the first split.

  • For Caltech-101 [79], the original does not dataset specifies any train/test splits. We have followed the approach used in [81]: This file defines datasets for 5 random splits of 25 training images per category, with 5 validation images per category and the remaining images used for testing.

  • For ImageNet, we took the last

    last images of the official tensorflow ImageNet split.

  • For Oxford-IIIT Pets, the valid set consists of 20 randomly selected images per class.

Information about the dataset are summarized in Table 10.

d.2 Transfer via linear classification

We follow the linear evaluation protocol of [44, 61, 8]

that we detail next for completeness. We train a regularized multinomial logistic regression classifier on top of the frozen representation, i.e., with frozen pretrained parameters and without re-computing batch-normalization statistics. In training and testing, we do not perform any image augmentations; images are resized to

pixels along the shorter side using bicubic resampling and then normalized with ImageNet statistics. Finally, we minimize the cross-entropy objective using LBFGS with -regularization, where we select the regularization parameters from a range of logarithmically-spaced values between and . After choosing the best-performing hyperparameters on the validation set, the model is retrained on combined training and validation images together, using the chosen parameters. The final accuracy is reported on the test set.

d.3 Transfer via fine-tuning

We follow the same fine-tuning protocol as in [29, 44, 63, 8] that we also detail for completeness. Specifically, we initialize the network with the parameters of the pretrained representation. At training time, we apply spatial transformation, i.e., random crops with resize to pixels and random flips. At test time, images are resized to pixels along the shorter side using bicubic resampling, after which a center crop is extracted. In both cases, we normalize the color channels by subtracting the average color and dividing by the standard deviation (computed on ImageNet), after applying the augmentations. We optimize the loss using SGD with Nesterov momentum for steps with a batch size of and with a momentum of . We set the momentum parameter for the batch normalization statistics to where is the number of steps per epoch. The learning rate and weight decay are selected respectively with a grid of seven logarithmically spaced learning rates between and , and logarithmically-spaced values of weight decay between and , as well as no weight decay. These values of weight decay are divided by the learning rate. After choosing the best-performing hyperparameters on the validation set, the model is retrained on combined training and validation images together, using the chosen parameters. The final accuracy is reported on the test set.

d.4 Implementation details for semantic segmentation

We use the same fully-convolutional network (FCN)-based [7] architecture as [9]. The backbone consists of the convolutional layers in ResNet-. The convolutions in the conv blocks use dilation

and stride

. This is followed by two extra convolutions with channels, each followed by batch normalization and ReLU activations, and a convolution for per-pixel classification. The dilation is set to in the two extra convolutions. The total stride is (FCN-s [7]).

We train on the train_aug2012 set and report results on val2012. Hyperparameters are selected on a images held-out validation set. We use a standard per-pixel softmax cross-entropy loss to train the FCN. Training is done with random scaling (by a ratio in ), cropping, and horizontal flipping. The crop size is . Inference is performed on the central crop. For training we use a batch size of and weight decay of . We select the base learning rate by sweeping across logarithmically spaced values between and . The learning rate is multiplied by at the -th and -th percentile of training. We train for iterations, and average the results on 5 seeds.

d.5 Implementation details for object detection

For object detection, we follow prior work on Pascal detection transfer [36, 20] wherever possible. We use a Faster R-CNN [69] detector with a R-C backbone with a frozen representation. The R-C backbone ends with the conv stage of a ResNet-, and the box prediction head consists of the conv stage (including global pooling). We preprocess the images by applying multi-scale augmentation (rescaling the image so its longest edge is between and pixels) but no other augmentation. We use an asynchronous SGD optimizer with workers and train for M steps. We used an initial learning rate of , which is reduced to at 1M steps and to at M steps.

d.6 Implementation details for depth estimation

For depth estimation, we follow the same protocol as in [70], and report its core components for completeness. We use a standard ResNet- backbone and feed the conv features into fast up-projection blocks with respective filter sizes , , , and

. We use a reverse Huber loss function for training 

[70, 82].

The original NYU Depth v frames of size are down-sampled by a factor and center-cropped to pixels. Input images are randomly horizontally flipped and the following color transformations are applied:

  • Grayscale with an application probability of .

  • Brightness with a maximum brightness difference of .

  • Saturation with a saturation factor randomly picked in the interval .

  • Hue with a hue adjustment factor randomly picked in the interval .

We train for steps with batch size , weight decay and learning rate (scaled linearly from the setup of [70] to account for the bigger batch size).

d.7 Further comparisons on PASCAL and NYU v2 Depth

For completeness,  Table 11 and 12 extends Table 4

with other published baselines which use comparable networks. We see that in almost all settings, BYOL outperforms these baselines, even when those baselines use more data or deeper models. One notable exception is RMS error for NYU Depth prediction, which is a metric that’s sensitive to outliers. The reason for this is unclear, but one possibility is that the network is producing higher-variance predictions due to being more confident about a test-set scene’s similarities with those in the training set.

Method AP mIoU
Supervised-IN [9]
RelPos [20], by [36] -
Multi-task [36] -
LocalAgg [83] -
MoCo [9]
MoCo + IG-1B [9]
CPC[29] -
SimCLR (repro)
BYOL (ours)
Table 11: Transfer results in semantic segmentation and object detection.
uses a larger model (ResNet-). uses an even larger model (ResNet-).
Higher better Lower better
Method pct.  pct.  pct.  rms rel
Supervised-IN [70]
RelPos [20], by [36]
Color [37], by [36]
Exemplar [42, 36]
Mot. Seg. [84], by [36]
Multi-task [36]
SimCLR (repro)
BYOL (ours)
Table 12: Transfer results on NYU v2 depth estimation.

Appendix E Pretraining on Places 365

To ascertain that BYOL learns good representations on other datasets, we applied our representation learning protocol on the scene recognition dataset Places-Standard [60] before performing linear evaluation. This dataset contains million training images and validation images with labels, making it roughly similar to ImageNet in scale. We reuse the exact same parameters as in Section 4 and train the representation for epochs, using BYOL and our SimCLR reproduction. Results for the linear evaluation setup (using the protocol of Section C.1 for ImageNet and Places365, and that of Appendix D on other datasets) are reported in Table 13.

Interestingly, the representation trained by using BYOL on Places365 (BYOL-PL) consistently outperforms that of SimCLR on the same dataset, but underperforms the BYOL representation trained on ImageNet (BYOL-IN) on all tasks except Places365 and SUN397 [66], another scene understanding dataset. Interestingly, all three unsupervised representation learning methods achieve a relatively high performance on the Places365 task; for comparison, reference [60] (in its linked repository) reports a top-1 accuracy of for a ResNet-v2 trained from scratch using labels on this dataset.

Method Places365 ImageNet Food101 CIFAR10 CIFAR100 Birdsnap SUN397 Cars Aircraft DTD Pets Caltech-101 Flowers
BYOL-IN
BYOL-PL
SimCLR-PL
Table 13: Transfer learning results (linear evaluation, ResNet-) from Places365 (PL). For comparison purposes, we also report the results from BYOL trained on ImageNet (BYOL-IN).

Appendix F Additional ablation results

To extend on the above results, we provide additional ablations obtained using the same experimental setup as in Section 5, i.e., 300 epochs, averaged over seeds with the initial learning rate set to , the batch size to , the weight decay to and the base target decay rate to

unless specified otherwise. Confidence intervals correspond to the half-difference between the maximum and minimum score of these seeds; we omit them for half-differences lower than

accuracy points.

f.1 Architecture settings

Table 14 shows the influence of projector and predictor architecture on BYOL. We examine the effect of different depths for both the projector and predictor, as well as the effect of the projection size. We do not apply a ReLU activation nor a batch normalization on the final linear layer of our MLPs such that a depth of corresponds to a linear layer. Using the default projector and predictor of depth yields the best performance.

Proj. 
depth
Pred. 
depth
Top- Top-
(a) Projector and predictor depth (i.e. the number of Linear layers).

aaaaa Projector 
output dim
Top- Top-
(b) Projection dimension.

Table 14: Effect of architectural settings where top- and top- accuracies are reported in %.

Table 14(a) shows the influence of the initial learning rate on BYOL. Note that the optimal value depends on the number of training epochs. Table 14(b) displays the influence of the weight decay on BYOL.

Learning
rate Top- Top-
(a) Base learning rate.
Weight decay
coefficient
Top- Top-
(b) Weight decay.

aaaaa

Table 15: Effect of learning rate and weight decay. We note that BYOL’s performance is quite robust within a range of hyperparameters.

f.2 Batch size

We run a sweep over the batch size for both BYOL and our reproduction of SimCLR. As explained in Section 5, when reducing the batch size by a factor , we average gradients over consecutive steps and update the target network once every steps. We report in Table 16, the performance of both our reproduction of SimCLR and BYOL for batch sizes between (BYOL and SimCLR default) down to . We observe that the performance of SimCLR deteriorates faster than the one of BYOL which stays mostly constant for batch sizes larger than . We believe that the performance at batch size could match the performance of the large batch size with proper parameter tuning when accumulating the gradient. We think that the drop in performance at batch size in table 16 is mainly related to the ill behaviour of batch normalization at low batch sizes [85].

Batch Top- Top-
size BYOL (ours) SimCLR (repro) BYOL (ours) SimCLR (repro)
Table 16: Influence of the batch size.

f.3 Image augmentations

Table 17 compares the impact of individual image transformations on BYOL and SimCLR. BYOL is more resilient to changes of image augmentations across the board.

Top- Top-
Image augmentation BYOL (ours) SimCLR (repro) BYOL (ours) SimCLR (repro)
Baseline
Remove flip
Remove blur
Remove color (jittering and grayscale)
Remove color jittering
Remove grayscale
Remove blur in
Remove solarize in
Remove blur and solarize in
Crop only
Crop and flip only
Crop and color only
Crop and blur only
Table 17: Ablation on image transformations.
<
Loss weight Temperature Top- Top-