Self-supervised Knowledge Distillation Using Singular Value Decomposition

07/18/2018
by   Seung-Hyun Lee, et al.
0

To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To improve the quality of the transferred knowledge from T-DNN, we propose a new knowledge distillation using singular value decomposition (SVD). In addition, we define a knowledge transfer as a self-supervised task and suggest a way to continuously receive information from T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of the T-DNN can be up to 1.1% better than the T-DNN in terms of classification accuracy. Also assuming the same computational cost, our S-DNN outperforms the S-DNN driven by the state-of-the-art distillation with a performance advantage of 1.79%. code is available on https://github.com/sseung0703/SSKD_SVD.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/03/2022

Attention Distillation: self-supervised vision transformer students need more guidance

Self-supervised learning has been widely applied to train high-quality v...
research
07/29/2021

Hierarchical Self-supervised Augmented Knowledge Distillation

Knowledge distillation often involves how to define and transfer knowled...
research
11/21/2022

LHDR: HDR Reconstruction for Legacy Content using a Lightweight DNN

High dynamic range (HDR) image is widely-used in graphics and photograph...
research
05/23/2023

Transferring Learning Trajectories of Neural Networks

Training deep neural networks (DNNs) is computationally expensive, which...
research
10/10/2020

Distilling a Deep Neural Network into a Takagi-Sugeno-Kang Fuzzy Inference System

Deep neural networks (DNNs) demonstrate great success in classification ...
research
11/17/2020

Deep Serial Number: Computational Watermarking for DNN Intellectual Property Protection

In this paper, we introduce DSN (Deep Serial Number), a new watermarking...
research
05/10/2021

KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation

Knowledge distillation (KD) has recently emerged as an efficacious schem...

Please sign up or login with your details

Forgot password? Click here to reset