Learning from a Teacher using Unlabeled Data

11/13/2019
by   Gaurav Menghani, et al.
0

Knowledge distillation is a widely used technique for model compression. We posit that the teacher model used in a distillation setup, captures relationships between classes, that extend beyond the original dataset. We empirically show that a teacher model can transfer this knowledge to a student model even on an out-of-distribution dataset. Using this approach, we show promising results on MNIST, CIFAR-10, and Caltech-256 datasets using unlabeled image data from different sources. Our results are encouraging and help shed further light from the perspective of understanding knowledge distillation and utilizing unlabeled data to improve model quality.

READ FULL TEXT
research
04/01/2022

Unified and Effective Ensemble Knowledge Distillation

Ensemble knowledge distillation can extract knowledge from multiple teac...
research
03/21/2017

Knowledge distillation using unlabeled mismatched images

Current approaches for Knowledge Distillation (KD) either directly use t...
research
01/17/2022

Distillation from heterogeneous unlabeled collections

Compressing deep networks is essential to expand their range of applicat...
research
02/17/2023

Explicit and Implicit Knowledge Distillation via Unlabeled Data

Data-free knowledge distillation is a challenging model lightweight task...
research
08/20/2022

Effectiveness of Function Matching in Driving Scene Recognition

Knowledge distillation is an effective approach for training compact rec...
research
11/03/2021

LTD: Low Temperature Distillation for Robust Adversarial Training

Adversarial training has been widely used to enhance the robustness of t...
research
03/28/2022

Knowledge Distillation: Bad Models Can Be Good Role Models

Large neural networks trained in the overparameterized regime are able t...

Please sign up or login with your details

Forgot password? Click here to reset