Weight Distillation: Transferring the Knowledge in Neural Network Parameters

09/19/2020
by   Ye Lin, et al.
0

Knowledge distillation has been proven to be effective in model acceleration and compression. It allows a small network to learn to generalize in the same way as a large network. Recent successes in pre-training suggest the effectiveness of transferring model parameters. Inspired by this, we investigate methods of model acceleration and compression in another line of research. We propose Weight Distillation to transfer the knowledge in the large network parameters through a parameter generator. Our experiments on WMT16 En-Ro, NIST12 Zh-En, and WMT14 En-De machine translation tasks show that weight distillation can train a small network that is 1.88 2.94x faster than the large network but with competitive performance. With the same sized small network, weight distillation can outperform knowledge distillation by 0.51 1.82 BLEU points.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 6

page 7

page 9

page 10

06/09/2020

Knowledge Distillation: A Survey

In recent years, deep neural networks have been very successful in the f...
06/25/2016

Sequence-Level Knowledge Distillation

Neural machine translation (NMT) offers a novel alternative formulation ...
01/07/2022

Microdosing: Knowledge Distillation for GAN based Compression

Recently, significant progress has been made in learned image and video ...
07/31/2022

Chinese grammatical error correction based on knowledge distillation

In view of the poor robustness of existing Chinese grammatical error cor...
11/11/2020

Real-Time Decentralized knowledge Transfer at the Edge

Proliferation of edge networks creates islands of learning agents workin...
08/25/2020

Discriminability Distillation in Group Representation Learning

Learning group representation is a commonly concerned issue in tasks whe...
12/01/2016

In Teacher We Trust: Learning Compressed Models for Pedestrian Detection

Deep convolutional neural networks continue to advance the state-of-the-...