Log In Sign Up

Collaborative Group Learning

by   Shaoxiong Feng, et al.

Collaborative learning has successfully applied knowledge transfer to guiding a pool of small student networks towards robust local minima. However, previous approaches typically struggle with drastically aggravated student homogenization and rapidly growing computational complexity when the number of students rises. In this paper, we propose Collaborative Group Learning, an efficient framework that aims to maximize student population without sacrificing generalization performance and computational efficiency. First, each student is established by randomly routing on a modular neural network, which is not only parameter-efficient but also facilitates flexible knowledge communication between students due to random levels of representation sharing and branching. Second, to resist homogenization and further reduce the computational cost, students first compose diverse feature sets by exploiting the inductive bias from sub-sets of training data, and then aggregate and distill supplementary knowledge by choosing a random sub-group of students at each time step. Empirical evaluations on both image and text tasks indicate that our method significantly outperforms various state-of-the-art collaborative approaches whilst enhancing computational efficiency.


page 1

page 2

page 3

page 4


Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation

Neural dialogue models suffer from low-quality responses when interacted...

AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models

Knowledge distillation (KD) methods compress large models into smaller s...

Collaborative Learning for Deep Neural Networks

We introduce collaborative learning in which multiple classifier heads o...

Impact of Bias on School Admissions and Targeted Interventions

There is an inherent problem in the way students are evaluated - be it s...

Exploring Student Representation For Neural Cognitive Diagnosis

Cognitive diagnosis, the goal of which is to obtain the proficiency leve...

Modeling Knowledge Acquisition from Multiple Learning Resource Types

Students acquire knowledge as they interact with a variety of learning m...

Quantitative analysis of approaches to group marking

Group work, where students work on projects to overcome challenges toget...