Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

02/27/2020
by   Xiang Deng, et al.
0

Substantial efforts have been made on improving the generalization abilities of deep neural networks (DNNs) in order to obtain better performances without introducing more parameters. On the other hand, meta-learning approaches exhibit powerful generalization on new tasks in few-shot learning. Intuitively, few-shot learning is more challenging than the standard supervised learning as each target class only has a very few or no training samples. The natural question that arises is whether the meta-learning idea can be used for improving the generalization of DNNs on the standard supervised learning. In this paper, we propose a novel meta-learning based training procedure (MLTP) for DNNs and demonstrate that the meta-learning idea can indeed improve the generalization abilities of DNNs. MLTP simulates the meta-training process by considering a batch of training samples as a task. The key idea is that the gradient descent step for improving the current task performance should also improve a new task performance, which is ignored by the current standard procedure for training neural networks. MLTP also benefits from all the existing training techniques such as dropout, weight decay, and batch normalization. We evaluate MLTP by training a variety of small and large neural networks on three benchmark datasets, i.e., CIFAR-10, CIFAR-100, and Tiny ImageNet. The experimental results show a consistently improved generalization performance on all the DNNs with different sizes, which verifies the promise of MLTP and demonstrates that the meta-learning idea is indeed able to improve the generalization of DNNs on the standard supervised learning.

READ FULL TEXT
research
12/06/2018

Meta-Transfer Learning for Few-Shot Learning

Meta-learning has been proposed as a framework to address the challengin...
research
07/09/2020

Few Is Enough: Task-Augmented Active Meta-Learning for Brain Cell Classification

Deep Neural Networks (or DNNs) must constantly cope with distribution ch...
research
03/06/2020

TaskNorm: Rethinking Batch Normalization for Meta-Learning

Modern meta-learning approaches for image classification rely on increas...
research
05/15/2019

LGM-Net: Learning to Generate Matching Networks for Few-Shot Learning

In this work, we propose a novel meta-learning approach for few-shot cla...
research
10/14/2022

Neural Routing in Meta Learning

Meta-learning often referred to as learning-to-learn is a promising noti...
research
06/20/2021

Task Attended Meta-Learning for Few-Shot Learning

Meta-learning (ML) has emerged as a promising direction in learning mode...
research
10/12/2022

A Unified Framework with Meta-dropout for Few-shot Learning

Conventional training of deep neural networks usually requires a substan...

Please sign up or login with your details

Forgot password? Click here to reset