Meta-Learning Parameterized First-Order Optimizers using Differentiable Convex Optimization

03/29/2023
by   Tanmay Gautam, et al.
0

Conventional optimization methods in machine learning and controls rely heavily on first-order update rules. Selecting the right method and hyperparameters for a particular task often involves trial-and-error or practitioner intuition, motivating the field of meta-learning. We generalize a broad family of preexisting update rules by proposing a meta-learning framework in which the inner loop optimization step involves solving a differentiable convex optimization (DCO). We illustrate the theoretical appeal of this approach by showing that it enables one-step optimization of a family of linear least squares problems, given that the meta-learner has sufficient exposure to similar tasks. Various instantiations of the DCO update rule are compared to conventional optimizers on a range of illustrative experimental settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/21/2020

Gradient-EM Bayesian Meta-learning

Bayesian meta-learning enables robust and fast adaptation to new tasks w...
research
07/14/2023

Multiplicative update rules for accelerating deep learning training and increasing robustness

Even nowadays, where Deep Learning (DL) has achieved state-of-the-art pe...
research
04/10/2021

Meta-Learning Bidirectional Update Rules

In this paper, we introduce a new type of generalized neural network whe...
research
03/16/2023

Arbitrary Order Meta-Learning with Simple Population-Based Evolution

Meta-learning, the notion of learning to learn, enables learning systems...
research
06/06/2019

Adaptive Gradient-Based Meta-Learning Methods

We build a theoretical framework for understanding practical meta-learni...
research
04/25/2022

Meta-AF: Meta-Learning for Adaptive Filters

Adaptive filtering algorithms are pervasive throughout modern society an...
research
07/30/2020

Bayesian Optimization for Developmental Robotics with Meta-Learning by Parameters Bounds Reduction

In robotics, methods and softwares usually require optimizations of hype...

Please sign up or login with your details

Forgot password? Click here to reset