DeepAI AI Chat
Log In Sign Up

Weighted Ensemble Self-Supervised Learning

by   Yangjun Ruan, et al.

Ensembling has proven to be a powerful technique for boosting model performance, uncertainty estimation, and robustness in supervised learning. Advances in self-supervised learning (SSL) enable leveraging large unlabeled corpora for state-of-the-art few-shot and supervised learning performance. In this paper, we explore how ensemble methods can improve recent SSL techniques by developing a framework that permits data-dependent weighted cross-entropy losses. We refrain from ensembling the representation backbone; this choice yields an efficient ensemble method that incurs a small training cost and requires no architectural changes or computational overhead to downstream evaluation. The effectiveness of our method is demonstrated with two state-of-the-art SSL methods, DINO (Caron et al., 2021) and MSN (Assran et al., 2022). Our method outperforms both in multiple evaluation metrics on ImageNet-1K, particularly in the few-shot setting. We explore several weighting schemes and find that those which increase the diversity of ensemble heads lead to better downstream evaluation results. Thorough experiments yield improved prior art baselines which our method still surpasses; e.g., our overall improvement with MSN ViT-B/16 is 3.9 p.p. for 1-shot learning.


page 25

page 26


Shot in the Dark: Few-Shot Learning with No Base-Class Labels

Few-shot learning aims to learn classifiers for new objects from a small...

Boosting Supervision with Self-Supervision for Few-shot Learning

We present a technique to improve the transferability of deep representa...

Charting the Right Manifold: Manifold Mixup for Few-shot Learning

Few-shot learning algorithms aim to learn model parameters capable of ad...

How Well Do Self-Supervised Methods Perform in Cross-Domain Few-Shot Learning?

Cross-domain few-shot learning (CDFSL) remains a largely unsolved proble...

Improved baselines for vision-language pre-training

Contrastive learning has emerged as an efficient framework to learn mult...

Multi-Pretext Attention Network for Few-shot Learning with Self-supervision

Few-shot learning is an interesting and challenging study, which enables...

Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers

As deep learning methods form a critical part in commercially important ...