Can a powerful neural network be a teacher for a weaker neural network?

05/01/2020
by   Nicola Landro, et al.
28

The transfer learning technique is widely used to learning in one context and applying it to another, i.e. the capacity to apply acquired knowledge and skills to new situations. But is it possible to transfer the learning from a deep neural network to a weaker neural network? Is it possible to improve the performance of a weak neural network using the knowledge acquired by a more powerful neural network? In this work, during the training process of a weak network, we add a loss function that minimizes the distance between the features previously learned from a strong neural network with the features that the weak network must try to learn. To demonstrate the effectiveness and robustness of our approach, we conducted a large number of experiments using three known datasets and demonstrated that a weak neural network can increase its performance if its learning process is driven by a more powerful neural network.

READ FULL TEXT

page 2

page 8

page 9

page 11

page 12

research
06/20/2022

A Neural Network Based Method with Transfer Learning for Genetic Data Analysis

Transfer learning has emerged as a powerful technique in many applicatio...
research
05/18/2015

Recurrent Neural Network Training with Dark Knowledge Transfer

Recurrent neural networks (RNNs), particularly long short-term memory (L...
research
10/15/2019

Transfer Learning for Algorithm Recommendation

Meta-Learning is a subarea of Machine Learning that aims to take advanta...
research
06/01/2017

Deep Mutual Learning

Model distillation is an effective and widely used technique to transfer...
research
03/01/2018

Knowledge Transfer with Jacobian Matching

Classical distillation methods transfer representations from a "teacher"...
research
03/20/2019

Representative Datasets: The Perceptron Case

One of the main drawbacks of the practical use of neural networks is the...
research
10/24/2017

Improving Accuracy of Nonparametric Transfer Learning via Vector Segmentation

Transfer learning using deep neural networks as feature extractors has b...

Please sign up or login with your details

Forgot password? Click here to reset