DeepAI
Log In Sign Up

Decoupling Adaptation from Modeling with Meta-Optimizers for Meta Learning

10/30/2019
by   Sébastien M. R. Arnold, et al.
11

Meta-learning methods, most notably Model-Agnostic Meta-Learning or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks. The mechanism behind their success, however, is poorly understood. We begin this work with an experimental analysis of MAML, finding that deep models are crucial for its success, even given sets of simple tasks where a linear model would suffice on any individual task. Furthermore, on image-recognition tasks, we find that the early layers of MAML-trained models learn task-invariant features, while later layers are used for adaptation, providing further evidence that these models require greater capacity than is strictly necessary for their individual tasks. Following our findings, we propose a method which enables better use of model capacity at inference time by separating the adaptation aspect of meta-learning into parameters that are only used for adaptation but are not part of the forward model. We find that our approach enables more effective meta-learning in smaller models, which are suitably sized for the individual tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/05/2018

The effects of negative adaptation in Model-Agnostic Meta-Learning

The capacity of meta-learning algorithms to quickly adapt to a variety o...
11/14/2022

Meta-Learning of Neural State-Space Models Using Data From Similar Systems

Deep neural state-space models (SSMs) provide a powerful tool for modeli...
10/07/2022

Learning to Learn and Sample BRDFs

We propose a method to accelerate the joint process of physically acquir...
03/17/2021

Meta-learning of Pooling Layers for Character Recognition

In convolutional neural network-based character recognition, pooling lay...
02/21/2021

Fast On-Device Adaptation for Spiking Neural Networks via Online-Within-Online Meta-Learning

Spiking Neural Networks (SNNs) have recently gained popularity as machin...
02/25/2016

Meta-learning within Projective Simulation

Learning models of artificial intelligence can nowadays perform very wel...
09/26/2019

Fast and Effective Adaptation of Facial Action Unit Detection Deep Model

Detecting facial action units (AU) is one of the fundamental steps in au...

Code Repositories

kfo

Code release for "When MAML Can Adapt Fast and How to Assist When It Cannot", AISTATS 2021.


view repo