Blockwise Self-Supervised Learning at Scale

02/03/2023
by   Shoaib Ahmed Siddiqui, et al.
4

Current state-of-the-art deep networks are all powered by backpropagation. In this paper, we explore alternatives to full backpropagation in the form of blockwise learning rules, leveraging the latest developments in self-supervised learning. We show that a blockwise pretraining procedure consisting of training independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins' loss function at each block performs almost as well as end-to-end backpropagation on ImageNet: a linear probe trained on top of our blockwise pretrained model obtains a top-1 classification accuracy of 70.48 below the accuracy of an end-to-end pretrained network (71.57 perform extensive experiments to understand the impact of different components within our method and explore a variety of adaptations of self-supervised learning to the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules to large networks, with implications ranging from hardware design to neuroscience.

READ FULL TEXT

page 3

page 5

page 17

research
09/21/2023

A Study of Forward-Forward Algorithm for Self-Supervised Learning

Self-supervised representation learning has seen remarkable progress in ...
research
09/26/2022

End-to-End Lyrics Recognition with Self-supervised Learning

Lyrics recognition is an important task in music processing. Despite tra...
research
10/09/2021

An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition

Self-supervised pretraining on speech data has achieved a lot of progres...
research
06/13/2020

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-su...
research
04/24/2023

A Cookbook of Self-Supervised Learning

Self-supervised learning, dubbed the dark matter of intelligence, is a p...
research
04/23/2023

End-to-End Feasible Optimization Proxies for Large-Scale Economic Dispatch

The paper proposes a novel End-to-End Learning and Repair (E2ELR) archit...
research
07/22/2022

Scale dependant layer for self-supervised nuclei encoding

Recent developments in self-supervised learning give us the possibility ...

Please sign up or login with your details

Forgot password? Click here to reset