DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning

07/13/2022
by   Shaoru Wang, et al.
0

Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications, it costs a huge computation burden to pretrain multiple networks of various sizes one by one. In this paper, we propose Discriminative-SSL-based Slimmable Pretrained Networks (DSPNet), which can be trained at once and then slimmed to multiple sub-networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource budgets. Specifically, we extend the idea of slimmable networks to a discriminative SSL paradigm, by integrating SSL and knowledge distillation gracefully. We show comparable or improved performance of DSPNet on ImageNet to the networks individually pretrained one by one under the linear evaluation and semi-supervised evaluation protocols, while reducing large training cost. The pretrained models also generalize well on downstream detection and segmentation tasks. Code will be made public.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/17/2021

Self-Supervised Learning for Binary Networks by Joint Classifier Training

Despite the great success of self-supervised learning with large floatin...
research
08/12/2021

AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

Transformer-based pretrained language models (T-PTLMs) have achieved gre...
research
02/07/2021

Self-supervised driven consistency training for annotation efficient histopathology image analysis

Training a neural network with a large labeled dataset is still a domina...
research
07/12/2022

Synergistic Self-supervised and Quantization Learning

With the success of self-supervised learning (SSL), it has become a main...
research
01/23/2023

A Simple Recipe for Competitive Low-compute Self supervised Vision Models

Self-supervised methods in vision have been mostly focused on large arch...
research
09/05/2023

Probabilistic Self-supervised Learning via Scoring Rules Minimization

In this paper, we propose a novel probabilistic self-supervised learning...
research
01/22/2023

Unifying Synergies between Self-supervised Learning and Dynamic Computation

Self-supervised learning (SSL) approaches have made major strides forwar...

Please sign up or login with your details

Forgot password? Click here to reset