Self-supervised Knowledge Distillation for Few-shot Learning

06/17/2020
by   Jathushan Rajasegaran, et al.
77

Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. Recent works [7, 41] show that simply learning a good feature embedding can outperform more sophisticated meta-learning and metric learning algorithms for few-shot learning. In this paper, we propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. We follow a two-stage learning process: First, we train a neural network to maximize the entropy of the feature embedding, thus creating an optimal output manifold using a self-supervised auxiliary loss. In the second stage, we minimize the entropy on feature embedding by bringing self-supervised twins together, while constraining the manifold with student-teacher distillation. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods, with further gains achieved by our second stage distillation process. Our codes are available at: https://github.com/brjathu/SKD.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/25/2020

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

The focus of recent meta-learning research has been on the development o...
research
03/01/2021

Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning

In many real-world problems, collecting a large number of labeled sample...
research
07/28/2019

Charting the Right Manifold: Manifold Mixup for Few-shot Learning

Few-shot learning algorithms aim to learn model parameters capable of ad...
research
09/17/2020

S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning

Deep Metric Learning (DML) provides a crucial tool for visual similarity...
research
03/25/2023

Supervised Masked Knowledge Distillation for Few-Shot Transformers

Vision Transformers (ViTs) emerge to achieve impressive performance on m...
research
11/03/2022

Robust Few-shot Learning Without Using any Adversarial Samples

The high cost of acquiring and annotating samples has made the `few-shot...
research
06/04/2022

Guided Deep Metric Learning

Deep Metric Learning (DML) methods have been proven relevant for visual ...

Please sign up or login with your details

Forgot password? Click here to reset