Intra-Model Collaborative Learning of Neural Networks

05/20/2021
by   Shijie Fang, et al.
0

Recently, collaborative learning proposed by Song and Chai has achieved remarkable improvements in image classification tasks by simultaneously training multiple classifier heads. However, huge memory footprints required by such multi-head structures may hinder the training of large-capacity baseline models. The natural question is how to achieve collaborative learning within a single network without duplicating any modules. In this paper, we propose four ways of collaborative learning among different parts of a single network with negligible engineering efforts. To improve the robustness of the network, we leverage the consistency of the output layer and intermediate layers for training under the collaborative learning framework. Besides, the similarity of intermediate representation and convolution kernel is also introduced to reduce the reduce redundant in a neural network. Compared to the method of Song and Chai, our framework further considers the collaboration inside a single model and takes smaller overhead. Extensive experiments on Cifar-10, Cifar-100, ImageNet32 and STL-10 corroborate the effectiveness of these four ways separately while combining them leads to further improvements. In particular, test errors on the STL-10 dataset are decreased by 9.28% and 5.45% for ResNet-18 and VGG-16 respectively. Moreover, our method is proven to be robust to label noise with experiments on Cifar-10 dataset. For example, our method has 3.53% higher performance under 50% noise ratio setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2018

Collaborative Learning for Deep Neural Networks

We introduce collaborative learning in which multiple classifier heads o...
research
09/12/2023

Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning

Deepfake has recently raised a plethora of societal concerns over its po...
research
01/08/2019

Learning with Collaborative Neural Network Group by Reflection

For the present engineering of neural systems, the preparing of extensiv...
research
05/24/2022

Multi-Agent Collaborative Inference via DNN Decoupling: Intermediate Feature Compression and Edge Learning

Recently, deploying deep neural network (DNN) models via collaborative i...
research
06/05/2019

Visual Confusion Label Tree For Image Classification

Convolution neural network models are widely used in image classificatio...
research
06/11/2021

Collaborative Multidisciplinary Design Optimization with Neural Networks

The design of complex engineering systems leads to solving very large op...
research
05/12/2020

Modularizing Deep Learning via Pairwise Learning With Kernels

By redefining the conventional notions of layers, we present an alternat...

Please sign up or login with your details

Forgot password? Click here to reset