Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers

05/28/2019
by   Jingwen Ye, et al.
0

Many well-trained Convolutional Neural Network(CNN) models have now been released online by developers for the sake of effortless reproducing. In this paper, we treat such pre-trained networks as teachers and explore how to learn a target student network for customized tasks, using multiple teachers that handle different tasks. We assume no human-labelled annotations are available, and each teacher model can be either single- or multi-task network, where the former is a degenerated case of the latter. The student model, depending on the customized tasks, learns the related knowledge filtered from the multiple teachers, and eventually masters the complete or a subset of expertise from all teachers. To this end, we adopt a layer-wise training strategy, which entangles the student's network block to be learned with the corresponding teachers. As demonstrated on several benchmarks, the learned student network achieves very promising results, even outperforming the teachers on the customized tasks.

READ FULL TEXT
research
08/20/2019

Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation

A massive number of well-trained deep networks have been released by dev...
research
04/23/2019

Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More

In this paper, we investigate a novel deep-model reusing task. Our goal ...
research
03/20/2020

Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

Recent advances in deep learning have provided procedures for learning o...
research
07/27/2022

Federated Selective Aggregation for Knowledge Amalgamation

In this paper, we explore a new knowledge-amalgamation problem, termed F...
research
06/24/2019

Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

An increasing number of well-trained deep networks have been released on...
research
11/07/2018

Amalgamating Knowledge towards Comprehensive Classification

With the rapid development of deep learning, there have been an unpreced...
research
06/16/2022

Learning to Teach Fairness-aware Deep Multi-task Learning

Fairness-aware learning mainly focuses on single task learning (STL). Th...

Please sign up or login with your details

Forgot password? Click here to reset