Deep Ensemble Collaborative Learning by using Knowledge-transfer Graph for Fine-grained Object Classification

03/27/2021
by   Naoki Okamoto, et al.
11

Mutual learning, in which multiple networks learn by sharing their knowledge, improves the performance of each network. However, the performance of ensembles of networks that have undergone mutual learning does not improve significantly from that of normal ensembles without mutual learning, even though the performance of each network has improved significantly. This may be due to the relationship between the knowledge in mutual learning and the individuality of the networks in the ensemble. In this study, we propose an ensemble method using knowledge transfer to improve the accuracy of ensembles by introducing a loss design that promotes diversity among networks in mutual learning. We use an attention map as knowledge, which represents the probability distribution and information in the middle layer of a network. There are many ways to combine networks and loss designs for knowledge transfer methods. Therefore, we use the automatic optimization of knowledge-transfer graphs to consider a variety of knowledge-transfer methods by graphically representing conventional mutual-learning and distillation methods and optimizing each element through hyperparameter search. The proposed method consists of a mechanism for constructing an ensemble in a knowledge-transfer graph, attention loss, and a loss design that promotes diversity among networks. We explore optimal ensemble learning by optimizing a knowledge-transfer graph to maximize ensemble accuracy. From exploration of graphs and evaluation experiments using the datasets of Stanford Dogs, Stanford Cars, and CUB-200-2011, we confirm that the proposed method is more accurate than a conventional ensemble method.

READ FULL TEXT

page 7

page 8

research
09/09/2020

Diversified Mutual Learning for Deep Metric Learning

Mutual learning is an ensemble training strategy to improve generalizati...
research
09/10/2019

Knowledge Transfer Graph for Deep Collaborative Learning

We propose Deep Collaborative Learning (DCL), which is a method that inc...
research
12/26/2021

Efficient Diversity-Driven Ensemble for Deep Neural Networks

The ensemble of deep neural networks has been shown, both theoretically ...
research
10/30/2019

When does Diversity Help Generalization in Classification Ensembles?

Ensembles, as a widely used and effective technique in the machine learn...
research
06/01/2017

Deep Mutual Learning

Model distillation is an effective and widely used technique to transfer...
research
06/27/2022

Transfer learning for ensembles: reducing computation time and keeping the diversity

Transferring a deep neural network trained on one problem to another req...
research
10/18/2021

Noise-Resilient Ensemble Learning using Evidence Accumulation Clustering

Ensemble Learning methods combine multiple algorithms performing the sam...

Please sign up or login with your details

Forgot password? Click here to reset