-
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
We propose an algorithm for meta-learning that is model-agnostic, in the...
03/09/2017 ∙ by Chelsea Finn, et al. ∙
0
∙
share
read it
-
Provable Meta-Learning of Linear Representations
Meta-learning, or learning-to-learn, seeks to design algorithms that can...
02/26/2020 ∙ by Nilesh Tripuraneni, et al. ∙
28
∙
share
read it
-
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
Learning to learn is a powerful paradigm for enabling models to learn fr...
10/31/2017 ∙ by Chelsea Finn, et al. ∙
0
∙
share
read it
-
Online gradient-based mixtures for transfer modulation in meta-learning
Learning-to-learn or meta-learning leverages data-driven inductive bias ...
12/14/2018 ∙ by Ghassen Jerfel, et al. ∙
0
∙
share
read it
-
Towards Understanding Generalization in Gradient-Based Meta-Learning
In this work we study generalization of neural networks in gradient-base...
07/16/2019 ∙ by Simon Guiroy, et al. ∙
9
∙
share
read it
-
Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes
We consider stochastic gradient descent (SGD) for least-squares regressi...
05/25/2018 ∙ by Loucas Pillaud-Vivien, et al. ∙
0
∙
share
read it
-
Meta-learning with differentiable closed-form solvers
Adapting deep networks to new concepts from few examples is extremely ch...
05/21/2018 ∙ by Luca Bertinetto, et al. ∙
0
∙
share
read it
Why Does MAML Outperform ERM? An Optimization Perspective
10/27/2020 ∙ by Liam Collins, et al. ∙
0
∙
share
Model-Agnostic Meta-Learning (MAML) has demonstrated widespread success in training models that can quickly adapt to new tasks via one or few stochastic gradient descent steps. However, the MAML objective is significantly more difficult to optimize compared to standard Empirical Risk Minimization (ERM), and little is understood about how much MAML improves over ERM in terms of the fast adaptability of their solutions in various scenarios. We analytically address this issue in a linear regression setting consisting of a mixture of easy and hard tasks, where hardness is determined by the number of gradient steps required to solve the task. Specifically, we prove that for Ω(d_eff) labelled test samples (for gradient-based fine-tuning) where d_eff is the effective dimension of the problem, in order for MAML to achieve substantial gain over ERM, the optimal solutions of the hard tasks must be closely packed together with the center far from the center of the easy task optimal solutions. We show that these insights also apply in a low-dimensional feature space when both MAML and ERM learn a representation of the tasks, which reduces the effective problem dimension. Further, our few-shot image classification experiments suggest that our results generalize beyond linear regression.
READ FULL TEXT
Comments
There are no comments yet.