DeepAI AI Chat
Log In Sign Up

Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks

by   Daiki Chijiwa, et al.

Few-shot learning for neural networks (NNs) is an important problem that aims to train NNs with a few data. The main challenge is how to avoid overfitting since over-parameterized NNs can easily overfit to such small dataset. Previous work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning, which learns how to learn from a few data by using various tasks. On the other hand, one conventional approach to avoid overfitting is restricting hypothesis spaces by endowing sparse NN structures like convolution layers in computer vision. However, although such manually-designed sparse structures are sample-efficient for sufficiently large datasets, they are still insufficient for few-shot learning. Then the following questions naturally arise: (1) Can we find sparse structures effective for few-shot learning by meta-learning? (2) What benefits will it bring in terms of meta-generalization? In this work, we propose a novel meta-learning approach, called Meta-ticket, to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. We empirically validated that Meta-ticket successfully discover sparse subnetworks that can learn specialized features for each given task. Due to this task-wise adaptation ability, Meta-ticket achieves superior meta-generalization compared to MAML-based methods especially with large NNs.


Meta-Learning with Network Pruning

Meta-learning is a powerful paradigm for few-shot learning. Although wit...

Meta-Meta-Classification for One-Shot Learning

We present a new approach, called meta-meta-classification, to learning ...

Meta-Learning Neural Bloom Filters

There has been a recent trend in training neural networks to replace dat...

L2AE-D: Learning to Aggregate Embeddings for Few-shot Learning with Meta-level Dropout

Few-shot learning focuses on learning a new visual concept with very lim...

Few-shot Learning for Decoding Brain Signals

Few-shot learning consists in addressing data-thrifty (inductive few-sho...

Few-Shot Learning with Per-Sample Rich Supervision

Learning with few samples is a major challenge for parameter-rich models...

A Unified Framework with Meta-dropout for Few-shot Learning

Conventional training of deep neural networks usually requires a substan...