MAML and ANIL Provably Learn Representations

by   Liam Collins, et al.

Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks. However, the mechanics of GBML have remained largely mysterious from a theoretical perspective. In this paper, we prove that two well-known GBML methods, MAML and ANIL, as well as their first-order approximations, are capable of learning common representation among a set of given tasks. Specifically, in the well-known multi-task linear representation learning setting, they are able to recover the ground-truth representation at an exponentially fast rate. Moreover, our analysis illuminates that the driving force causing MAML and ANIL to recover the underlying representation is that they adapt the final layer of their model, which harnesses the underlying task diversity to improve the representation in all directions of interest. To the best of our knowledge, these are the first results to show that MAML and/or ANIL learn expressive representations and to rigorously explain why they do so.


page 1

page 2

page 3

page 4


Provable Meta-Learning of Linear Representations

Meta-learning, or learning-to-learn, seeks to design algorithms that can...

Few-Shot Learning via Learning the Representation, Provably

This paper studies few-shot learning via representation learning, where ...

The Effect of Diversity in Meta-Learning

Few-shot learning aims to learn representations that can tackle novel ta...

Conditional Meta-Learning of Linear Representations

Standard meta-learning for representation learning aims to find a common...

Improving Password Guessing via Representation Learning

Learning useful representations from unstructured data is one of the cor...

A Meta-Learning Approach for Graph Representation Learning in Multi-Task Settings

Graph Neural Networks (GNNs) are a framework for graph representation le...

Representation Learning Beyond Linear Prediction Functions

Recent papers on the theory of representation learning has shown the imp...