Multi-step Estimation for Gradient-based Meta-learning

06/08/2020
by   Jin-Hwa Kim, et al.
9

Gradient-based meta-learning approaches have been successful in few-shot learning, transfer learning, and a wide range of other domains. Despite its efficacy and simplicity, the burden of calculating the Hessian matrix with large memory footprints is the critical challenge in large-scale applications. To tackle this issue, we propose a simple yet straightforward method to reduce the cost by reusing the same gradient in a window of inner steps. We describe the dynamics of the multi-step estimation in the Lagrangian formalism and discuss how to reduce evaluating second-order derivatives estimating the dynamics. To validate our method, we experiment on meta-transfer learning and few-shot learning tasks for multiple settings. The experiment on meta-transfer emphasizes the applicability of training meta-networks, where other approximations are limited. For few-shot learning, we evaluate time and memory complexities compared with popular baselines. We show that our method significantly reduces training time and memory usage, maintaining competitive accuracies, or even outperforming in some cases.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/06/2018

Meta-Transfer Learning for Few-Shot Learning

Meta-learning has been proposed as a framework to address the challengin...
research
04/06/2021

Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

Meta and transfer learning are two successful families of approaches to ...
research
07/31/2023

MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning

Equipping a deep model the abaility of few-shot learning, i.e., learning...
research
04/26/2022

Meta-free few-shot learning via representation learning with weight averaging

Recent studies on few-shot classification using transfer learning pose c...
research
06/19/2021

EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization

Gradient-based meta-learning and hyperparameter optimization have seen s...
research
10/13/2021

ES-Based Jacobian Enables Faster Bilevel Optimization

Bilevel optimization (BO) has arisen as a powerful tool for solving many...
research
07/02/2021

Memory Efficient Meta-Learning with Large Images

Meta learning approaches to few-shot classification are computationally ...

Please sign up or login with your details

Forgot password? Click here to reset