Deep Mutual Learning

06/01/2017
by   Ying Zhang, et al.
0

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, that is better suited to low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy where, rather than one way transfer between a static pre-defined teacher and a student, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on CIFAR-100 recognition and Market-1501 person re-identification benchmarks. Surprisingly, it is revealed that no prior powerful teacher network is necessary -- mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/21/2021

Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression

Knowledge distillation (KD) is an effective model compression technique ...
research
11/26/2018

ExpandNets: Exploiting Linear Redundancy to Train Small Networks

While very deep networks can achieve great performance, they are ill-sui...
research
02/10/2020

Subclass Distillation

After a large "teacher" neural network has been trained on labeled data,...
research
08/18/2020

Knowledge Transfer via Dense Cross-Layer Mutual-Distillation

Knowledge Distillation (KD) based methods adopt the one-way Knowledge Tr...
research
01/23/2019

Distillation Strategies for Proximal Policy Optimization

Vision-based deep reinforcement learning (RL), similar to deep learning,...
research
05/01/2020

Can a powerful neural network be a teacher for a weaker neural network?

The transfer learning technique is widely used to learning in one contex...
research
03/27/2021

Deep Ensemble Collaborative Learning by using Knowledge-transfer Graph for Fine-grained Object Classification

Mutual learning, in which multiple networks learn by sharing their knowl...

Please sign up or login with your details

Forgot password? Click here to reset