Small-Group Learning, with Application to Neural Architecture Search

12/23/2020
by   Xuefeng Du, et al.
0

Small-group learning is a broadly used methodology in human learning and shows great effectiveness in improving learning outcomes: a small group of students work together towards the same learning objective, where they express their understanding of a topic to their peers, compare their ideas, and help each other to trouble-shoot problems. We are interested in investigating whether this powerful learning technique can be borrowed from humans to improve the learning abilities of machines. We propose a novel learning approach called small-group learning (SGL). In our approach, each learner uses its intermediately trained model to generate a pseudo-labeled dataset and re-trains its model using pseudo-labeled datasets generated by other learners. We propose a multi-level optimization framework to formulate SGL which involves three learning stages: learners train their network weights independently; learners train their network weights collaboratively via mutual pseudo-labeling; learners improve their architectures by minimizing validation losses. We develop an efficient algorithm to solve the SGL problem. We apply our approach to neural architecture search and achieve significant improvement on CIFAR-100, CIFAR-10, and ImageNet.

READ FULL TEXT
12/23/2020

Learning by Self-Explanation, with Application to Neural Architecture Search

Learning by self-explanation, where students explain a learned topic to ...
12/09/2020

Skillearn: Machine Learning Inspired by Humans' Learning Skills

Humans, as the most powerful learners on the planet, have accumulated a ...
11/30/2020

Learning by Passing Tests, with Application to Neural Architecture Search

Learning through tests is a broadly used methodology in human learning a...
12/01/2021

Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search

Learning from mistakes is an effective learning approach widely used in ...
12/28/2020

Learning by Ignoring

Learning by ignoring, which identifies less important things and exclude...
03/12/2021

Interleaving Learning, with Application to Neural Architecture Search

Interleaving learning is a human learning technique where a learner inte...
09/22/2021

Learning by Examples Based on Multi-level Optimization

Learning by examples, which learns to solve a new problem by looking int...