Amalgamating Knowledge towards Comprehensive Classification

11/07/2018
by   Chengchao Shen, et al.
0

With the rapid development of deep learning, there have been an unprecedentedly large number of trained deep network models available online. Reusing such trained models can significantly reduce the cost of training the new models from scratch, if not infeasible at all as the annotations used for the training original networks are often unavailable to public. We propose in this paper to study a new model-reusing task, which we term as knowledge amalgamation. Given multiple trained teacher networks, each of which specializes in a different classification problem, the goal of knowledge amalgamation is to learn a lightweight student model capable of handling the comprehensive classification. We assume no other annotations except the outputs from the teacher models are available, and thus focus on extracting and amalgamating knowledge from the multiple teachers. To this end, we propose a pilot two-step strategy to tackle the knowledge amalgamation task, by learning first the compact feature representations from teachers and then the network parameters in a layer-wise manner so as to build the student model. We apply this approach to four public datasets and obtain very encouraging results: even without any human annotation, the obtained student model is competent to handle the comprehensive classification task and in most cases outperforms the teachers in individual sub-tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/24/2019

Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

An increasing number of well-trained deep networks have been released on...
research
08/20/2019

Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation

A massive number of well-trained deep networks have been released by dev...
research
04/23/2019

Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More

In this paper, we investigate a novel deep-model reusing task. Our goal ...
research
05/28/2019

Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers

Many well-trained Convolutional Neural Network(CNN) models have now been...
research
04/30/2021

Distilling EEG Representations via Capsules for Affective Computing

Affective computing with Electroencephalogram (EEG) is a challenging tas...
research
05/16/2023

Low-complexity deep learning frameworks for acoustic scene classification using teacher-student scheme and multiple spectrograms

In this technical report, a low-complexity deep learning system for acou...
research
08/17/2023

Learning Through Guidance: Knowledge Distillation for Endoscopic Image Classification

Endoscopy plays a major role in identifying any underlying abnormalities...

Please sign up or login with your details

Forgot password? Click here to reset