byol-pytorch
Usable Implementation of "Bootstrap Your Own Latent" self-supervised learning, from Deepmind, in Pytorch
view repo
Contrastive approaches to self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing the same from different data points (negative pairs). However, recent approaches like BYOL and SimSiam, show remarkable performance without negative pairs, raising a fundamental theoretical question: how can SSL with only positive pairs avoid representational collapse? We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our analysis yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Furthermore, motivated by our theory we propose a novel approach that directly sets the predictor based on the statistics of its inputs. In the case of linear predictors, our approach outperforms gradient training of the predictor by 5% and on ImageNet it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm. Code is released in https://github.com/facebookresearch/luckmatters/tree/master/ssl.
READ FULL TEXT
Recently, contrastive learning has achieved great results in self-superv...
read it
We propose a novel theoretical framework to understand self-supervised
l...
read it
Self-supervised representation learning has witnessed significant leaps
...
read it
In this paper, we propose a method, named EqCo (Equivalent Rules for
Con...
read it
Contrastive self-supervised learning (CSL) is an approach to learn usefu...
read it
Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach ...
read it
Self-supervised contrastive learning between pairs of multiple views of ...
read it
Comments
There are no comments yet.