Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

04/06/2021
by   Vincent Dumoulin, et al.
8

Meta and transfer learning are two successful families of approaches to few-shot learning. Despite highly related goals, state-of-the-art advances in each family are measured largely in isolation of each other. As a result of diverging evaluation norms, a direct or thorough comparison of different approaches is challenging. To bridge this gap, we perform a cross-family study of the best transfer and meta learners on both a large-scale meta-learning benchmark (Meta-Dataset, MD), and a transfer learning benchmark (Visual Task Adaptation Benchmark, VTAB). We find that, on average, large-scale transfer methods (Big Transfer, BiT) outperform competing approaches on MD, even when trained only on ImageNet. In contrast, meta-learning approaches struggle to compete on VTAB when trained and validated on MD. However, BiT is not without limitations, and pushing for scale does not improve performance on highly out-of-distribution MD tasks. In performing this study, we reveal a number of discrepancies in evaluation norms and study some of these in light of the performance gap. We hope that this work facilitates sharing of insights from each community, and accelerates progress on few-shot learning.

READ FULL TEXT

page 6

page 18

page 19

12/16/2019

A New Benchmark for Evaluation of Cross-Domain Few-Shot Learning

Recent progress on few-shot learning has largely re-lied on annotated da...
07/02/2021

Memory Efficient Meta-Learning with Large Images

Meta learning approaches to few-shot classification are computationally ...
07/15/2021

FLEX: Unifying Evaluation for Few-Shot NLP

Few-shot NLP research is highly active, yet conducted in disjoint resear...
06/08/2020

Multi-step Estimation for Gradient-based Meta-learning

Gradient-based meta-learning approaches have been successful in few-shot...
08/02/2022

The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence

Recently, it has been observed that a transfer learning solution might b...
10/11/2017

Neural Program Meta-Induction

Most recently proposed methods for Neural Program Induction work under t...
07/22/2019

Domain-Specific Priors and Meta Learning for Low-shot First-Person Action Recognition

The lack of large-scale real datasets with annotationsmakes transfer lea...